2026

The performance of large language models is tied to the quality and organization of their training data. However, existing data metrics emphasize surface-level properties, such as sequence length or model-based quality scores, and provide limited insight into how the internal organization of data relates to optimization. We study the structural fabric of training data, defined as the pattern of semantic dependencies within a sequence. We quantify this property using dependency density, a normalized measure of effective segment-level dependencies. Across data length, domain, and format, we find that dependency density varies with data configuration: code shows higher dependency density than natural language within long-context data, and instruction-tuning mappings show higher dependency density than raw intra-input modeling. We show that coherent structural fabric is associated with a lower gradient effective rank, indicating a more concentrated optimization geometry with reduced gradient interference. We observe that this low-rank optimization regime is associated with a more stable intrinsic dimension, preserving a compact representation manifold that may mitigate representation noise and support downstream performance. These results provide a geometric perspective on how data structure relates to LLM training, linking data configuration, optimization dynamics, and model behavior through the lens of structural fabric.