Leakage-Aware Evaluation of Brazilian Clinical Notes for ICU Mortality Prediction: A Study on the BRATECA Dataset

Felipe André Bach Alves, Heloísa Oss Boll, Mariana Recamonde-Mendoza


Abstract
Early prediction of ICU mortality can support clinical decisions, but retrospective notes can contain future-event cues that cause leakage in clinical NLP. We evaluate leakage-aware modeling of Brazilian Portuguese ICU notes for mortality prediction using BRATECA. We compare 24-hour restriction, regex and LLM leakage auditing, neural and TF-IDF representations, and unrestricted-note baselines. In the 24-hour setting, models achieved AUROC 0.78-0.83; unrestricted notes reached around 0.95, indicating outcome-related information in late documentation. Regex found leakage signals in 21.13% of the notes, and the LLM audit identified additional semantic cases. Results highlight the need for temporal control and leakage auditing in clinical NLP.
Anthology ID:
2026.stil-1.2
Volume:
Proceedings of the 17th Brazilian Symposium in Information and Human Language Technology
Month:
October
Year:
2026
Address:
Cuiabá, Mato Grosso, Brazil
Editors:
Bryan Khelven da Silva Barbosa, Aline Paes, Ariani Di Felippo
Venue:
STIL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
14–25
Language:
URL:
https://aclanthology.org/2026.stil-1.2/
DOI:
10.5753/stil.2026.26535
Bibkey:
Cite (ACL):
Felipe André Bach Alves, Heloísa Oss Boll, and Mariana Recamonde-Mendoza. 2026. Leakage-Aware Evaluation of Brazilian Clinical Notes for ICU Mortality Prediction: A Study on the BRATECA Dataset. In Proceedings of the 17th Brazilian Symposium in Information and Human Language Technology, pages 14–25, Cuiabá, Mato Grosso, Brazil. Association for Computational Linguistics.
Cite (Informal):
Leakage-Aware Evaluation of Brazilian Clinical Notes for ICU Mortality Prediction: A Study on the BRATECA Dataset (Alves et al., STIL 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.stil-1.2.pdf