Caio Alves Caldeira
Author directory2026
On the Role of Supporting Evidence in LLM Reasoning: A Causal Analysis of Legal QA with Activation Patching
Dalmo Buzato | Fernanda Sathler Rocha Franco | Caio Alves Caldeira | Isabela Carolina Ribeiro Santos | Evandro L. T. P. Cunha | Pedro Calais | Wagner Meira Jr. | Anisio Lacerda
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Dalmo Buzato | Fernanda Sathler Rocha Franco | Caio Alves Caldeira | Isabela Carolina Ribeiro Santos | Evandro L. T. P. Cunha | Pedro Calais | Wagner Meira Jr. | Anisio Lacerda
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
As large language models (LLMs) are increasingly considered for high-stakes legal applications, a central question for trust and auditability is whether their outputs reflect genuine reasoning over the supplied legal materials, or merely surface statistical patterns absorbed during pre-training. We address this question at the mechanistic level by applying activation patching, a causal-mediation technique, to four open-source multilingual LLMs on two bar-exam datasets: U.S. MBE and Brazilian OAB. For each multiple-choice question we contrast a gold supporting passage with an irrelevant-but-plausible counterfactual. We then surgically replace residual-stream activations across layers and token regions and measure recovery of the logit difference. Gold passages exert a measurable but model- and linguistic-dependent causal effect; integration concentrates in the upper layers (a late-binding signature); and the size of the effect is strongly mediated by the linguistic and discourse structure of the passage itself, not just its relevance. We introduce a four-cluster taxonomy that operationalises these regimes and identifies cases where retrieved evidence is causally decisive, inert, or actively harmful.