Yassine Thlija

Author directory

2026

Faithfulness assesses whether a model’s explanation reflects what actually drove its answer, which is different from assessing answer correctness. But does faithfulness also apply in expert domains such as law, where the relevant factors that decide the answer, also called concepts, are diffuse and only implicitly expressed? We measure faithfulness in two expert-annotated legal reasoning multiple-choice question answering (QA) datasets and show that existing methods are not tailor-made for the legal domain. We therefore introduce a detailed, per-concept measure which reveals how each concept is being relied on and expressed by a model. We test our method with a small, open-weight LLM and find that faithfulness is generally low on legal datasets and that simple supervised fine-tuning lowers faithfulness even further, despite giving plausible legal explanations.