Ana Pinto
Author directory2026
Auditing Bias Detection Under Source Shift: A Source-Aware Benchmark and Evaluation Protocol
Ana Pinto | Alvaro Figueira
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
Ana Pinto | Alvaro Figueira
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
Bias-detection benchmarks routinely report accuracies above 90%, but a high number can reflect where a sentence came from as much as whether it contains a stereotype or an unfair claim about a group. We treat this as an evaluation problem and propose an integrated auditing protocol that makes source confounding measurable: an LLM-assisted label audit with cross-family verification, pair-aware splitting with role tracking, source-aware evaluation with a source-only baseline on the actual test split, counterfactual discrimination, and an artifact analysis of edited text. We apply it to a three-source corpus of 10,304 sentences that we audit and release. The audit reduces the global source-only majority baseline from 95.0% to 55.4%, and the same protocol shows that the real-only test split still carries a 77.6% source-only baseline, so confounding is reduced and measured but far from eliminated. Against that baseline, fine-tuned BERT and GPT-2 gain roughly 18 pp and outperform attention-derived models; four augmentation variants that look ordered in the mean fall within seed variance under paired testing; and a real-vs-edited probe separates Gemini counterfactuals from real text, although the signal disappears once the gold label is held fixed and has no measurable accuracy effect. Applied to CrowS-Pairs, the label audit flags 33.2% of the supposedly neutral members as still containing a group-level stereotype. The main contribution is the protocol, which exposes confounding that standard pipelines would report as results.
2006
Corpógrafo V3 - From Terminological Aid to Semi-automatic Knowledge Engineering
Luís Sarmento | Belinda Maia | Diana Santos | Ana Pinto | Luís Cabral
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)
Luís Sarmento | Belinda Maia | Diana Santos | Ana Pinto | Luís Cabral
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)
In this paper we will present Corpógrafo, a mature web-based environment for working with corpora, for terminology extraction, and for ontology development. We will explain Corpógrafo’s workflow and describe the most important information extraction methods used, namely its term extraction, and definition / semantic relations identification procedures. We will describe current Corpógrafo users and present a brief overview of the XML format currently used to export terminology databases. Finally, we present future improvements for this tool.