Lathish Balaji Baskaran

Author directory

2026

Large language models do not just answer questions in dialogue; they are also expected to make the right conversational move. When a user apologizes, thanks the assistant, corrects an earlier claim, expresses frustration, or sets a preference, the reply carries a pragmatic obligation. We call failures to meet that obligation uptake failures, and model statements that imply unsupported memory or future behavior void commitments. We introduce a 300-item benchmark of user-assistant exchanges across six speech-act families, three difficulty tiers, and both synthetic and adapted-from-real-conversation items. This benchmark is intended as a curated core set: small enough for careful agreement analysis, but structured so that additional balanced batches can be added under the same ontology. On the current core set, we find strong annotation agreement (κ_family = 1.00, κ_act = 0.978). We also present SpeakFix, a lightweight post-generation repair layer that detects likely contract violations and fixes them with a prepend-not-replace strategy. Across GPT-4o, Claude Sonnet 4.6, and GPT-5.2, baseline uptake accuracy ranges from 71.0% to 77.3%, while void commitment rates range from 8% to 9%. SpeakFix raises uptake accuracy to 87.0–93.3% and reduces void commitments to 0%; the paired uptake gains are statistically significant for all three models (p < 10^-10). In a two-reviewer manual evaluation of 54 repaired failure cases (108 total judgments), 70.4% of repairs are rated better than the original, 28.7% equivalent, and 0.9% worse; helpfulness and naturalness are preserved in 100% and 99.1% of judgments, respectively. A separate blinded study with four independent, non-author raters is more critical: it confirms the repairs are at least as helpful as the original, but finds that the minimal prepend-style edits are often judged less natural, so reliable uptake gains do not yet translate into uniformly preferred rewrites. Taken together, the results suggest that speech-act contract violations are common enough to measure and often reducible with lightweight repair.