Alfred Shen
Author directory2026
The Flip-Test: A Behavioral Diagnostic for Multi-Agent Debate Failures
Julia Hu | Alfred Shen | Kumar Lakshmipathi
Proceedings of the Seventh Workshop on Insights from Negative Results in NLP
Julia Hu | Alfred Shen | Kumar Lakshmipathi
Proceedings of the Seventh Workshop on Insights from Negative Results in NLP
Multi-agent debate has been proposed to improve language model reasoning through adversarial interaction, yet empirical studies report that debate underperforms simpler baselines on open-ended reasoning (Choi et al., 2025). We propose a behavioral diagnostic—the flip-test—that instruments debate at the example level and measures how often, and in which direction, agents change their answers between initial and final rounds. On 2,417 MuSiQue examples with each of Llama 3.1 8B and Ministral 3 8B, flips are common (65–70% of debates). Our central result conditions on opportunity: a flip destroys an available correct answer∼2.4×more often than it creates one (48.9% vs. 20.3% on Llama, 44.2% vs. 18.0% on Ministral; p 10−20). Unlike a raw directional split, this measure is invariant to how many correct answers were available, and it is statistically indistinguishable across the two model families. Three instruction-level interventions on a 100-example held-out set produce no improvement, suggesting simple prompt-level patches are insufficient. We separate this behavioral finding from mechanism: direct opponent-adoption is low (∼10%), so the asymmetry is consistent with sycophancy but does not establish it. The signature is also scale-dependent—on three 70B-class models the flip rate roughly halves and debate turns net-positive—placing the harmful regime in the 8B capacity band rather than in debate itself. We frame the flip-test as a cheap behavioural probe for locating that regime.