Fernanda Sathler Rocha Franco

Author directory

2026

As large language models (LLMs) are increasingly considered for high-stakes legal applications, a central question for trust and auditability is whether their outputs reflect genuine reasoning over the supplied legal materials, or merely surface statistical patterns absorbed during pre-training. We address this question at the mechanistic level by applying activation patching, a causal-mediation technique, to four open-source multilingual LLMs on two bar-exam datasets: U.S. MBE and Brazilian OAB. For each multiple-choice question we contrast a gold supporting passage with an irrelevant-but-plausible counterfactual. We then surgically replace residual-stream activations across layers and token regions and measure recovery of the logit difference. Gold passages exert a measurable but model- and linguistic-dependent causal effect; integration concentrates in the upper layers (a late-binding signature); and the size of the effect is strongly mediated by the linguistic and discourse structure of the passage itself, not just its relevance. We introduce a four-cluster taxonomy that operationalises these regimes and identifies cases where retrieved evidence is causally decisive, inert, or actively harmful.