Beyond Better Helpers: A Shared-Kernel Study of Symbolic Interfaces for Auditable LLM Algebra

Norio Kosaka, Genki Osada, Shinichi Higashino, Shuji Yamaguchi


Abstract
Large language models can delegate algebra to external tools, but improvements attributed to a structured tool interface may instead come from the symbolic helper code hidden behind that interface. We isolate these factors on polynomial systems by holding the model, tasks, backend, and symbolic kernel fixed. Using Qwen3-8B, we compare text-only reasoning, unrestricted Python/SymPy generation, generated Python calling the same helper functions as the structured system, and a typed diagnosis–solve–check interface. On 36 tasks, the typed interface obtains 25/36 correct answers, compared with 14/36 for matched-helper code, 17/36 for raw SymPy, and 16/36 for text only. It also produces 23/36 automatically verified-correct answers versus 4/36 for the matched-helper control, and contains no execution-error runs versus 29/36. The largest separation occurs on exact finite-set tasks, where the typed interface solves 8/12 systems and both code conditions solve 0/12. Access to packaged helper functionality alone therefore does not reproduce the typed condition’s performance. The typed interface also removes the need to generate imports, API calls, control flow, and result-handling code, so the comparison measures end-to-end interface usability rather than mathematical reasoning or schema syntax in isolation. We interpret the result as a deployed-interface effect for one model and a development-informed task suite.
Anthology ID:
2026.mathnlp-main.1
Volume:
The Proceedings of the 4th Workshop on Mathematical Natural Language Processing (MathNLP 2026)
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Marco Valentino, Deborah Ferreira, Mokanarangan Thayaparan, Leonardo Ranaldi, Jo-Ku Cheng, Andre Freitas
Venues:
MathNLP | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–12
Language:
URL:
https://aclanthology.org/2026.mathnlp-main.1/
DOI:
Bibkey:
Cite (ACL):
Norio Kosaka, Genki Osada, Shinichi Higashino, and Shuji Yamaguchi. 2026. Beyond Better Helpers: A Shared-Kernel Study of Symbolic Interfaces for Auditable LLM Algebra. In The Proceedings of the 4th Workshop on Mathematical Natural Language Processing (MathNLP 2026), pages 1–12, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Beyond Better Helpers: A Shared-Kernel Study of Symbolic Interfaces for Auditable LLM Algebra (Kosaka et al., MathNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.mathnlp-main.1.pdf