Joost Visser
Author directory2026
Privacy-Preserving Information Extraction with Local LLMs: A Comparative Study on Dutch Debt Collection Letters
Beyza Celep | Natalia Amat-Lefort | Joost Visser
Proceedings of the Fifteenth Language Resources and Evaluation Conference
Beyza Celep | Natalia Amat-Lefort | Joost Visser
Proceedings of the Fifteenth Language Resources and Evaluation Conference
For individuals in financial distress, understanding debt collection letters is critical. These documents are often unstructured, use complex legal language, and contain highly sensitive personal data. Automating information extraction is essential for assisting caseworkers, who currently perform this task manually; a slow and error-prone process. The sensitive nature of this data requires efficient, privacy-preserving, locally-deployed solutions. This paper compares the feasibility of various local NLP models for this task. We evaluated a feature-engineered Conditional Random Field (CRF), a fine-tuned spaCy NER model, and several Large Language Models (LLMs) (1.1B to 14B parameters) on a new synthetic dataset of 1,000 Dutch debt letters. Models were compared using accuracy (F1-score) and deployment metrics (CPU runtime, memory usage). Our results show a clear performance-resource trade-off. Lightweight CRF and spaCy models efficiently extracted structured data but failed in many critical unstructured fields. In contrast, LLM performance scaled directly with model size. The 14B DeepSeek model achieved the highest accuracy (95.2% average F1), successfully handling all field types. In conclusion, larger local LLMs are the most viable solution for accurate, private document processing. Alternatively, a hybrid approach using lightweight models for structured data and LLMs only for complex, unstructured fields, would also be adequate.
ReleaseEval: A Benchmark for Evaluating Language Models in Automated Release Note Generation
Qianru Meng | Zhaochun Ren | Joost Visser
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Qianru Meng | Zhaochun Ren | Joost Visser
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Automated release note generation aims to reduce the cost of documenting frequent software updates. Existing benchmarks mainly rely on commit messages, limiting evaluation of how models use broader software-change evidence. We introduce ReleaseEval, a reproducible and openly licensed multi-granularity benchmark with 94,987 release notes from 3,369 repositories across six programming languages. Each release is aligned with commit messages, commit tree structures, and raw code diffs, supporting three task settings: commit2sum, tree2sum, and diff2sum. Automatic and human evaluations show that models benefit from lightweight commit-tree structure, while raw diffs remain challenging due to length and noise. These findings position ReleaseEval as a multi-granularity benchmark for studying release note generation. We release the dataset and code at https://github.com/Changanyuexia/rn_benchmark/
Search