Xiaotian Tan
Author directory2026
Directional Structurality: A Lightweight Proxy for Reasoning Data Quality
Ze Chen | Kaixuan Liu | Haomai Zhang | Jiaxuan Zou | Zhuo Chen | Xiaotian Tan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Ze Chen | Kaixuan Liu | Haomai Zhang | Jiaxuan Zou | Zhuo Chen | Xiaotian Tan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Reasoning-oriented language model training relies on traces that expose learnable dependencies, yet existing data-selection criteria use length, perplexity, difficulty, or judge-assigned quality. We argue that useful reasoning traces should also be sensitive to dependency order: intermediate steps are easier to encode in their natural premise-to-conclusion order but less predictable when dependency-revealing order is perturbed.
We introduce Local-DeltaAvg, a training-free data-selection metric. For each reasoning trace, we construct a semantically reversed counterpart that preserves problem content while perturbing dependency-revealing order. A fixed probe model scores both trajectories, and Local-DeltaAvg computes a paired local negative log-likelihood gap between the original and reversed contexts. The resulting probe-dependent score estimates the conditional encoding cost induced by order perturbation and identifies where this sensitivity is concentrated.
We evaluate Local-DeltaAvg as a relative ranking signal for reasoning-oriented continued-pretraining data selection, followed by an identical downstream SFT protocol. Under matched token budgets, it consistently outperforms length-, perplexity-, difficulty-, and judge-based selection across diverse reasoning benchmarks. Control analyses show that semantic reversal is essential and that high-gap regions align with annotated critical reasoning steps. These results suggest that dependency-order sensitivity is a lightweight and effective proxy for reasoning-data utility.
We introduce Local-DeltaAvg, a training-free data-selection metric. For each reasoning trace, we construct a semantically reversed counterpart that preserves problem content while perturbing dependency-revealing order. A fixed probe model scores both trajectories, and Local-DeltaAvg computes a paired local negative log-likelihood gap between the original and reversed contexts. The resulting probe-dependent score estimates the conditional encoding cost induced by order perturbation and identifies where this sensitivity is concentrated.
We evaluate Local-DeltaAvg as a relative ranking signal for reasoning-oriented continued-pretraining data selection, followed by an identical downstream SFT protocol. Under matched token budgets, it consistently outperforms length-, perplexity-, difficulty-, and judge-based selection across diverse reasoning benchmarks. Control analyses show that semantic reversal is essential and that high-gap regions align with annotated critical reasoning steps. These results suggest that dependency-order sensitivity is a lightweight and effective proxy for reasoning-data utility.