Enhancing Criminal Investigation Analysis with Summarization and Memory-based Retrieval-Augmented Generation: A Comprehensive Evaluation of Real Case Data

Mads Skipanes, Tollef Emil JÃ, rgensen, Kyle Porter, Gianluca Demartini, Sule Yildirim Yayilgan


Abstract
This study introduces KriRAG, a novel Retrieval-Augmented Generation (RAG) architecture designed to assist criminal investigators in analyzing information and overcoming the challenge of information overload. KriRAG structures and summarizes extensive document collections based on existing investigative queries, providing relevant document references and detailed answers for each query. Working with unstructured data from two homicide case files comprising approximately 3,700 documents and 13,000 pages, a comprehensive evaluation methodology is established, incorporating semantic retrieval, scoring, reasoning, and query response accuracy. The system’s outputs are evaluated against queries and answers provided by criminal investigators, demonstrating promising performance with 97.5% accuracy in relevance assessment and 77.5% accuracy for query responses. These findings provide a rigorous foundation for other query-oriented and open-ended retrieval applications. KriRAG is designed to run offline on limited hardware, ensuring sensitive data handling and on-device availability.
Anthology ID:
2025.coling-main.334
Volume:
Proceedings of the 31st International Conference on Computational Linguistics
Month:
January
Year:
2025
Address:
Abu Dhabi, UAE
Editors:
Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert
Venue:
COLING
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4993–5010
Language:
URL:
https://aclanthology.org/2025.coling-main.334/
DOI:
Bibkey:
Cite (ACL):
Mads Skipanes, Tollef Emil JÃ, rgensen, Kyle Porter, Gianluca Demartini, and Sule Yildirim Yayilgan. 2025. Enhancing Criminal Investigation Analysis with Summarization and Memory-based Retrieval-Augmented Generation: A Comprehensive Evaluation of Real Case Data. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4993–5010, Abu Dhabi, UAE. Association for Computational Linguistics.
Cite (Informal):
Enhancing Criminal Investigation Analysis with Summarization and Memory-based Retrieval-Augmented Generation: A Comprehensive Evaluation of Real Case Data (Skipanes et al., COLING 2025)
Copy Citation:
PDF:
https://aclanthology.org/2025.coling-main.334.pdf