Samer Aslan

Author directory

2026

Recent advances in large language models (LLMs) have significantly enhanced automated legal reasoning. Yet, it remains unclear whether their performance reflects genuine legal reasoning ability or artifacts of data contamination. We present a comprehensive empirical study of tax law reasoning approaches and implement a contamination detection protocol to rigorously assess LLM reliability. We show that performance can be inflated by contamination. Building on this analysis, we conduct a systematic evaluation, comparing monolithic LLMs with hybrid systems that translate statutory text into formal representations and delegate inference to symbolic solvers. We build a novel test suite designed to probe generalization to unseen documents via case and rule variations. Our findings indicate that legal reasoning is inherently compositional and that neuro-symbolic frameworks offer a more reliable and robust foundation for legal AI, as well as improved generalization to unobserved situations.
In-depth logical reasoning in domains such as legal analysis remains challenging for large language models (LLMs), exhibiting brittle and inconsistent performance on tasks requiring multi-step deductive inference. This limitation is expected, as LLMs primarily rely on learned distributional associations rather than performing explicit step-by-step logical inference through formal rule application. To address this gap, we propose a novel framework, called PercepLog, that incorporates a classical deductive reasoning algorithm, backward chaining, within an outer control loop, while utilizing LLMs for primitive operations such as substitution and unification. By following the backward chaining algorithm, the framework benefits from the interpretability and explainability of symbolic reasoning procedures while retaining the flexibility of LLM perception for processing natural-language legal text. We evaluate the proposed method on statutory reasoning tasks across a range of legal domains as well general logical reasoning. The results demonstrate that our approach enables more robust, explainable, and principled reasoning over unstructured legal text, where exhaustive formalization is often brittle and prone to hallucination. Our method yields average relative performance improvements of 58% over direct LLM reasoning and 16% over formal LLM-based translation and Prolog inference pipelines.