Alberto Aguzzi
Author directory2026
Paint It, Black: Evaluating LLMs for De-Identification of Multimodal Clinical Documents
Francesco Zangrillo | Alessio Cocchieri | Giacomo Frisoni | Davide Freddi | Luca Ragazzi | Alberto Aguzzi | Diego Raimondo | Renato Seracchioli | Gianluca Moro
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Francesco Zangrillo | Alessio Cocchieri | Giacomo Frisoni | Davide Freddi | Luca Ragazzi | Alberto Aguzzi | Diego Raimondo | Renato Seracchioli | Gianluca Moro
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Secondary use of clinical data requires removing Protected Health Information (PHI) under regulations such as HIPAA. While de-identification is well studied for text, clinical records are often structured PDFs, where PHI appears across complex layouts, tables, and headers. We introduce Paint-It-Black (PIB), the first benchmark for end-to-end, pixel-level PHI de-identification in clinical PDFs. PIB contains physician-validated synthetic documents with diverse layouts, document types, and controlled visual degradations, annotated with HIPAA-aligned labels and bounding boxes. We evaluate locally deployable Multimodal Large Language Models (MLLMs) and find that even the strongest remain below the reliability required for clinical use. A conventional OCR+NER baseline performs substantially worse than the best MLLMs, highlighting the inherently multimodal nature of clinical PHI de-identification.