Haokun Luo

Author directory

2026

Legal judgments may inadvertently contain errors, yet human appellate review faces immense pressure to improve efficiency. While current legal AI focuses on legal judgment prediction or document generation, judgment review is fundamentally a retrospective diagnostic task. In this paper, we introduce Appellate Review to assess large language models’ (LLMs) legal diagnostic reasoning underlying appellate review. We specifically target Chinese criminal cases, under the assumption that their highly structured logic, which maps explicit facts to strict statutory penalties, provides a verifiable ground truth. We decompose Appellate Review into three subtasks: error detection, error classification, and error correction. To support this, we construct AR-Bench, a controlled benchmark comprising 8,700 judgments with counterfactually introduced errors spanning charges, prison terms, and fines. Evaluations of 16 LLM variants reveal substantial limitations: models classify known errors relatively well but struggle with unguided detection and precise correction. Further analyses of a representative model show that it identifies explicit textual contradictions but fundamentally fails at holistic deductive reasoning and rule-based arithmetic.