Rajeev Thakur
Author directory2026
LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks
Chih-Hsuan Yang | Jingyan Jiang | Cheng-Hau Yang | Vikram Vasudevan | Huihuo Zheng | Venkatram Vishwanath | Rajeev Thakur
Findings of the Association for Computational Linguistics: EMNLP 2026
Chih-Hsuan Yang | Jingyan Jiang | Cheng-Hau Yang | Vikram Vasudevan | Huihuo Zheng | Venkatram Vishwanath | Rajeev Thakur
Findings of the Association for Computational Linguistics: EMNLP 2026
Multi-agent large language model (LLM) systems can improve reasoning by spending more computation, but deployment requires deciding when extra collaboration is worth its cost. We isolate this decision by running every problem under four protocols while holding the solver fixed within each setting: direct solving (Baseline), iterative self-correction (Single), planner–executor–reviewer collaboration (PER), and multi-agent deliberation (Broadcast). The primary benchmark comprises 4,181 competition-level math problems; paired robustness checks cover four benchmarks spanning competition math, biology, and broader science with two solver families. Across fixed policies, trained routers, and frozen LLM routers, conservative policies under-escalate, whereas higher-solve frozen routers often over-escalate. A post-answer, pre-collaboration gpt-oss-120b probe ranks Baseline failures with 0.8847 AUROC (4,151 parseable cases; 95% CI [0.8732, 0.8955]). The same score remains informative for predicting whether any collaboration helps (0.7683 AUPRC), but is much weaker for identifying PER- or Broadcast-specific value (0.1674 and 0.1041 AUPRC). Separately, the pre-answer self-confidence gate reaches 78.0% solve at 45K tokens, compared with 73.8% at 71.3K for a frozen gpt-oss-120b router and 92.4% for a retrospective fixed-order oracle. Across 10 paired model–condition settings, the oracle adds 23.2–58.3 points of retrospective coverage over Baseline, but protocol profiles vary by task. In the six settings with held-out router evaluations, oracle gaps remain 18.5–28.9 points. Confidence can therefore support initial escalation, while protocol-specific cost-aware routing remains unresolved.
Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translation
Le Chen | Nuo Xu | Winson Chen | Bin Lei | Pei-Hung Lin | Dunzhi Zhou | Rajeev Thakur | Caiwen Ding | Ali Jannesari | Chunhua Liao
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Le Chen | Nuo Xu | Winson Chen | Bin Lei | Pei-Hung Lin | Dunzhi Zhou | Rajeev Thakur | Caiwen Ding | Ali Jannesari | Chunhua Liao
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Large language models (LLMs) have shown remarkable capabilities in code translation, yet their performance deteriorates in low-resource programming domains such as Fortran and emerging frameworks like CUDA, where high-quality parallel data are scarce. We present an automated dataset generation pipeline featuring a dual-LLM Questioner–Solver design that incorporates external knowledge from compilers and runtime feedback. Beyond traditional source–target code pair datasets, our approach additionally generates (1) verified translations with unit tests for assessing functional consistency, and (2) multi-turn dialogues that capture the reasoning process behind translation refinement. Applied to Fortran→C++ and C++→CUDA, the pipeline yields 3.64k and 3.93k dialogues, respectively. Fine-tuning on this data yields dramatic improvements in functional correctness, boosting unit test success rates by over 56% on the challenging C++-to-CUDA task. We show that the generated data enables a 7B open-weight model to significantly outperform larger proprietary systems on key metrics like compilation success.