Lailah Denny
Author directory2026
Lost In Translation: Uneven Grounding Degradation Under Cross-Lingual Shift in LLM-Assisted Cyber Threat Intelligence
Lailah Denny
Proceedings of the 1st Workshop on Grounding Language Models: Learning Faithfully and Efficiently (GroundLM 2026)
Lailah Denny
Proceedings of the 1st Workshop on Grounding Language Models: Learning Faithfully and Efficiently (GroundLM 2026)
We introduce CrossCTI, a multilingual benchmark for evaluating retrieval-augmented generation (RAG) grounding faithfulness in open-source large language models applied to cyber threat intelligence (CTI) analysis. Existing CTI and forensic NLP benchmarks are predominantly English-centric, leaving a critical gap in understanding how grounding faithfulness degrades under cross-lingual distribution shift, a realistic condition in multinational security operations centers. CrossCTI translates 50 English CTI reports into Arabic, Chinese, Swahili, and Russian using NLLB-200 with entity-preserving masking, achieving 100% IOC preservation across languages. We construct 26 forensic question-answer pairs and evaluate each against all five language corpora and three open-source models (Qwen2.5-7B, Llama-3.1-8B, and Mistral-7B-v0.3), producing 390 evaluation instances, scored with token-level F1, character-level F1, and entailment-based attribution. Using a repeated-measures mixed-effects analysis that accounts for the fact that each of the 26 items is evaluated under all 15 model×language conditions, we find that cross-lingual degradation is severe, but uneven across metrics: language is the dominant, robust predictor of lexical (token-level) grounding quality (p<0.001), with no detectable effect of model identity on this metric. In contrast, model identity is a significant predictor of both character-level overlap and entailment-based attribution (p<0.01), including a specific interaction in one model (Mistral) that underperforms specifically in Swahili. Token-level F1 drops from 0.44 (English) to 0.03 (Arabic); gold-chunk retrieval recall drops from 0.962 (English) to 0.115 (Arabic); and the instance-level correlation between retrieval success and token-level grounding is r=0.60. Performance also improves from the top-k=1 to the top-k=3 retrieved passages before plateauing, with no consistent gains from the additional retrieved content. We further show that the entailment-based attribution metric is significantly elevated specifically for the model used to generate the benchmark’s own QA pairs, a synthetic-annotator confound that we quantify directly via within-item paired tests rather than only disclosing qualitatively. CrossCTI provides a reproducible framework for multilingual CTI grounding evaluation and a case study in how the apparent dominant factor, language or model, depends on which grounding metric is used.