Ruining Chen

Author directory

2026

Pretraining-data attribution typically assigns scalar scores to individual training examples. However, practical data curation involves retaining or removing coherent subsets, making additive pointwise assumptions mathematically restrictive. To address this, we formalize attribution as subset-level counterfactual utility prediction and propose GRASP (Geometry-aware Residual Alignment for Subset Prediction). GRASP combines first-order residual alignment with a low-dimensional train-train geometry term, derived from a smoothness lower bound. By explicitly modeling data interactions via compact 65-dimensional feature sketches, GRASP reduces upfront artifact construction time by nearly an order of magnitude compared to standard baselines. It efficiently scores 100,000 candidate subsets in just 5 seconds on a single CPU, completely bypassing the prohibitive cost of combinatorial retraining. Empirically, GRASP strictly dominates scalable baselines, more than doubling the average task-level subset-utility rank correlation on Linear Datamodeling Score (LDS) benchmarks, while fixed-compute diagnostics on Common Crawl further qualify its practical utility in large-scale curation.