Adrian Tan
Author directory2026
Absolute Error Is Not Enough: Ranking Model Performance Without Labels to Estimate Disparities at Scale
Veronica Rammouz | Aaron O. Gonzalez | Carlos Cruzportillo | Adrian Tan | Nicole Lang Beebe | Ke Yang | Anthony Rios
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Veronica Rammouz | Aaron O. Gonzalez | Carlos Cruzportillo | Adrian Tan | Nicole Lang Beebe | Ke Yang | Anthony Rios
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Estimating model performance without labeled data is important for deploying NLP systems across diverse domains. Prior work has proposed proxy measures based on distribution shift or predicted correctness, but it remains unclear whether these methods reliably preserve the ranking of domains, which is critical for model selection and risk triage. We study this problem from an ordinal perspective using a two-step framework that pairs a base classifier with an error predictor to estimate instance-level failures and disparities and aggregate them into label-free performance estimates. Across multiple sample sizes from the GeoOLID and Amazon Reviews datasets, large language model–based error predictors substantially outperform drift-based baselines in 𝜌 and 𝜏 rank correlations. We further show that lower absolute estimation error (RMSE) does not necessarily imply better ranking, highlighting the importance of evaluating ordinal consistency directly.
WARNING: This paper contains offensive and hateful speech in the appendix.
WARNING: This paper contains offensive and hateful speech in the appendix.
2025
Beyond Text-to-SQL for IoT Defense: A Comprehensive Framework for Querying and Classifying IoT Threats
Ryan Pavlich | Nima Ebadi | Richard Tarbell | Billy Linares | Adrian Tan | Rachael Humphreys | Jayanta Das | Rambod Ghandiparsi | Hannah Haley | Jerris George | Rocky Slavin | Kim-Kwang Raymond Choo | Glenn Dietrich | Anthony Rios
Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025)
Ryan Pavlich | Nima Ebadi | Richard Tarbell | Billy Linares | Adrian Tan | Rachael Humphreys | Jayanta Das | Rambod Ghandiparsi | Hannah Haley | Jerris George | Rocky Slavin | Kim-Kwang Raymond Choo | Glenn Dietrich | Anthony Rios
Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025)
Recognizing the promise of natural language interfaces to databases, prior studies have emphasized the development of text-to-SQL systems. Existing research has generally focused on generating SQL statements from text queries, and the broader challenge lies in inferring new information about the returned data. Our research makes two major contributions to address this gap. First, we introduce a novel Internet-of-Things (IoT) text-to-SQL dataset comprising 10,985 text-SQL pairs and 239,398 rows of network traffic activity. The dataset contains additional query types limited in prior text-to-SQL datasets, notably, temporal-related queries. Our dataset is sourced from a smart building’s IoT ecosystem exploring sensor read and network traffic data. Second, our dataset allows two-stage processing, where the returned data (network traffic) from a generated SQL can be categorized as malicious or not. Our results show that joint training to query and infer information about the data improves overall text-to-SQL performance, nearly matching that of substantially larger models. We also show that current large language models (e.g., GPT3.5) struggle to infer new information about returned data (i.e., they are bad at tabular data understanding), thus our dataset provides a novel test bed for integrating complex domain-specific reasoning into LLMs.
2024
Extracting Biomedical Entities from Noisy Audio Transcripts
Nima Ebadi | Kellen Morgan | Adrian Tan | Billy Linares | Sheri Osborn | Emma Majors | Jeremy Davis | Anthony Rios
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Nima Ebadi | Kellen Morgan | Adrian Tan | Billy Linares | Sheri Osborn | Emma Majors | Jeremy Davis | Anthony Rios
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Automatic Speech Recognition (ASR) technology is fundamental in transcribing spoken language into text, with considerable applications in the clinical realm, including streamlining medical transcription and integrating with Electronic Health Record (EHR) systems. Nevertheless, challenges persist, especially when transcriptions contain noise, leading to significant drops in performance when Natural Language Processing (NLP) models are applied. Named Entity Recognition (NER), an essential clinical task, is particularly affected by such noise, often termed the ASR-NLP gap. Prior works have primarily studied ASR’s efficiency in clean recordings, leaving a research gap concerning the performance in noisy environments. This paper introduces a novel dataset, BioASR-NER, designed to bridge the ASR-NLP gap in the biomedical domain, focusing on extracting adverse drug reactions and mentions of entities from the Brief Test of Adult Cognition by Telephone (BTACT) exam. Our dataset offers a comprehensive collection of almost 2,000 clean and noisy recordings. In addressing the noise challenge, we present an innovative transcript-cleaning method using GPT-4, investigating both zero-shot and few-shot methodologies. Our study further delves into an error analysis, shedding light on the types of errors in transcription software, corrections by GPT-4, and the challenges GPT-4 faces. This paper aims to foster improved understanding and potential solutions for the ASR-NLP gap, ultimately supporting enhanced healthcare documentation practices.
Search