Sebastien Kawada

Author directory

2026

Across self-consistency samples from an LLM, vote agreement tracks instance difficulty: on SemEval-2026 Task 4 (Narrative Story Similarity), supermajority cases (≥ 7/8 votes) resolve at 85% accuracy, split votes at 67%, and perfect ties at 61%, a monotone gradient that holds across the development set. We exploit this in CascadeMind, which routes eight Gemini 2.5 Flash votes by consensus, escalates split votes to additional sampling rounds, and falls through to a symbolic ensemble of theory-inspired narrative signals only on perfect ties (5% of cases). The system reached 72.75% on Track A test, placing 10th of 44 teams. Ablations show that the symbolic component contributes negligibly end-to-end and that nearly all gains come from confidence-aware routing. The takeaway is methodological: for narrative similarity, calibrating when to spend more compute on a hard instance matters more than adding auxiliary representations to reason about it. Code is available at https://github.com/chreia/CascadeMind-ACL.
Political evasion is difficult to detect because evasive answers often appear cooperative while avoiding concrete commitment. We present AsymVerify, a confidence-gated verification system for SemEval-2026 Task 6, a three-way classification of Clear Reply, Ambivalent, and Clear Non-Reply responses. AsymVerify scored 0.85 Macro F1 on the evaluation split (Deval, n=237), placing 2nd out of 41 teams on the official leaderboard. The system first classifies each question-answer pair, then selectively applies downgrade verification (CR/CNR → AMB) or upgrade verification (AMB → CR) to low-confidence predictions. Development analysis shows that errors concentrate at the Ambivalent boundary in both directions, motivating this asymmetric two-verifier design while confidence gating keeps additional inference cost low. On Ddev (n=308), AsymVerify with GLM-4.7 gains +17.1 Macro F1 over single-pass classification at 1.48 calls/example, and the upgrade verifier alone improves every tested LLM backend on Ddev by +6.8 to +15.2 Macro F1 over its single-pass baseline. Code is available at https://github.com/kaons-research/AsymVerify-ACL.
Questions in law, policy, the life sciences, and the humanities often require evidence from many sources, all of which retrieval must find. We prove that recall can be high even when few questions have all of their gold items retrieved, that is, are complete. We demonstrate that completeness strongly affects answer correctness. On HotpotQA questions where Quorum, our learned fusion of diverse retrievers, is the only retriever to retrieve all of the supporting paragraphs, our reader’s exact match rises from 0.212 to 0.553. We find that recall hides poor completeness. On Housing Statute QA questions that cite four or more statutes, Quorum’s top-ten results include an average of 39.3% of the gold items, yet are complete for only 2.3% of the questions. Recall exceeds completeness by 17- to 81-fold on this subset for all seven systems we evaluate. The candidate pools contain all of the gold items for 65% of these questions, meaning that the reranker causes more failures of completeness than the candidate retrieval. Quorum significantly outperforms the baseline, improving completeness@5 over dense retrieval from 0.453 to 0.670 on MuSiQue and accuracy on the agent-memory benchmark LongMemEval-V2 from 0.385 to 0.530. Benchmarks where multiple items must be retrieved should report completeness conditioned on the number of gold items.