KARMA: Karma-Aligned Reward Model Adaptation

Jared Scott, Jesse Roberts


Abstract
Human communication depends on implicit social signals where effectiveness is shaped by tone, context, and conversational norms rather than semantic content alone. We introduce KARMA (Karma-Aligned Reward Model Adaptation), a framework for LLM learning of context-sensitive conversational behavior from large-scale social media data. KARMA trains a reward model on Reddit conversations to predict response valuation conditioned on context, and uses this signal to fine-tune language models via reinforcement learning to improve performance on pragmatics-mediated tasks. Critically, we find that the highest performing reward model does not lead to better downstream model alignment: a reward model relying exclusively on conversational context was a worse predictor of Reddit karma but yielded substantially better downstream performance. We evaluate the effects of KARMA applied to a downstream model with and without direct exposure to the social media data. The resulting models show improved pragmatics-mediated behaviors with largely mitigated undesirable side effects.
Anthology ID:
2026.findings-emnlp.2243
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
48030–48039
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.2243/
DOI:
10.18653/2026.findings-emnlp.2243
Bibkey:
Cite (ACL):
Jared Scott and Jesse Roberts. 2026. KARMA: Karma-Aligned Reward Model Adaptation. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 48030–48039, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
KARMA: Karma-Aligned Reward Model Adaptation (Scott & Roberts, Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.2243.pdf