Yongzhen Guo
Author directory2026
MathDebugger: A Benchmark for Type-Aware Error Detection in Synthetic Math Data
Hao Liang | Meiyi Qiang | Yuying Li | Zefeng He | Xiaochen Ma | Ruichuan An | Yongzhen Guo | Zhengzhou Zhu | Bin Cui | Wentao Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Hao Liang | Meiyi Qiang | Yuying Li | Zefeng He | Xiaochen Ma | Ruichuan An | Yongzhen Guo | Zhengzhou Zhu | Bin Cui | Wentao Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Synthetic math data is widely used to scale LLM training, but its correctness is not guaranteed. We introduce MathDebugger, the first text-only, type-aware benchmark for jointly detecting *and* classifying errors in synthetic mathematical questions and answers. MathDebugger contains 2,000 correct questions, 2,000 erroneous questions (balanced across four fine-grained error types), and 2,000 annotated answers (610 erroneous, across three error types), each annotated with both a correctness label and an error-type label, with per-error-type Fleiss’ 𝜅 ranging from 0.69 to 0.88. On 14 SOTA LLMs (6 closed-source, 8 open-source, 1.5B–72B) and 3 process reward models, even advanced systems such as GPT-o3 and DeepSeek-R1 remain far from saturating the benchmark, particularly on fine-grained error-type detection; we further surface a *solving-vs-verifying* gap that causes math-tuned and long-chain-reasoning models to trail their general-purpose siblings. Error-type annotations also provide actionable supervision for correction, with gains that are significant under paired bootstrap (p<0.01). Code and data: [https://anonymous.4open.science/r/MathDebugger-7C4A/README.md](https://anonymous.4open.science/r/MathDebugger-7C4A/README.md)