Raj Firke
Author directory2026
The Correlation Mirage: Benchmark Dependence Collapses for Top-Performing LLMs
Raj Firke | Rajeswari Kannan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Raj Firke | Rajeswari Kannan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Standard approaches to benchmark redundancy rely on Spearman correlation, which implicitly treats dependence as uniform across the performance spectrum. Using copula-based dependence modeling on 4,497 models (779 deduplicated models; 714 unique base-model families) from the Open LLM Leaderboard v2, we show this assumption fails dramatically: benchmark pairs correlating at 𝜌S > 0.73 show near-zero or reversed ranking agreement among top-performing models, with conditional Spearman reductions of up to 97%—findings that hold after cluster-robust inference at the base-model-family level (≥96% bootstrap stability for two of three headline pairs, 𝛥BIC CIs entirely above zero). The mechanism is non-Gaussian dependence structure: 47% of benchmark pairs follow copula families with zero asymptotic tail dependence despite strong bulk association (𝛥BIC up to 585), confirmed by nonparametric validation requiring no parametric assumptions. We validate across BenchPress (316 benchmarks; signal strengthens with sample size) and kensho/WILD (27 benchmarks, 65 independent-organization models; 38% Frank prevalence), and demonstrate that no scalar dependence measure—Spearman, distance correlation, or mutual information—can detect the tail decomposition. In plain terms: a model ranked first on one benchmark but last on another is not unusual among frontier models—precisely the ones practitioners are deploying.
When Does Reasoning Age? Survival Analysis of Step-Level Error Hazard in LLM Chains
Raj Firke | Rajeswari Kannan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Raj Firke | Rajeswari Kannan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
We apply survival analysis to study how the probability of reasoning error evolves across steps in LLM chain-of-thought inference. Analyzing 18,969 chains across five benchmarks and seven models (3B–70B) under greedy decoding, we discover a near-universal aging effect—the conditional probability of reasoning error increasing with step depth—as measured by the Weibull shape parameter 𝜌 > 1 (indicating increasing hazard) in 34 of 35 dataset–model cells (binomial p = 10-9), with task-dependent magnitude ranging from ̄𝜌 = 1.35 (multi-domain QA) to ̄𝜌 = 2.36 (logical deduction). This finding is robust to informative censoring (IPCW adjustment: 34/35 cells retain 𝜌 > 1) and unobserved problem-level heterogeneity (shared frailty 𝜃 ≈ 0: 34/35 cells retain 𝜌 > 1). Compositional fan-in—the number of prior-step outputs consumed per step—shows a large but statistically suggestive rank correlation with aging magnitude (Spearman 𝜌s = 0.90, exact permutation p = 0.083, N = 5). We show that a survivorship bias masks these dynamics when raw error rates are used instead of the conditional hazard function. Held-out verification experiments confirm that late-weighted allocation (motivated by the aging finding) catches 94% of errors vs. 23% uniform on the strongest-aging task at budget K=1, while hazard-proportional allocation outperforms both baselines on moderately aging tasks (GSM8K, MathQA) at all budget levels. Our framework requires no training and provides actionable diagnostics for verification allocation.