Automatic metrics stand in for native-speaker evaluation in low-resource retrieval-augmented generation (RAG), so their validity determines what can be concluded about these systems. We test whether common metrics detect re- trieval and grounding failure in one fixed RAG pipeline across ten Indic languages. They do not merely miss the failure in Santali; they invert it. Hit@5 is 0.011, yet raw XLM-R BERTScore is the highest in the set at 0.902, changes little when retrieved evidence is re- moved, and assigns nearly the same score to matched and unrelated references. The cause is coverage: the scorer maps 39% of Santali tokens to <unk>, and a scorer whose vocab- ulary includes Ol Chiki restores discrimina- tion while LaBSE does not. Retrieval fails for the same reason. The gold article is indexed for 80% of Santali queries, but the retriever shares XLM-R’s tokenizer and collapses before reranking. Judge-based faithfulness and con- text precision, computed by a model with a dif- ferent tokenizer, flag the failure independently. A metric that cannot represent a script does not become noisy; it becomes constant and high. We argue that metric validity in low-resource languages must be established per language, and that evidence removal, reference shuffling, and the scorer’s <unk> rate can be important additions.

Clinical NLP increasingly relies on electronic health record (EHR) datato detect suicidal behaviors, treating clinical documentation as morereliable ground truth than social media. We argue that this framingobscures how EHR-based suicidality datasets encode a particularoperationalization of suicidality, shaped by who authors the data,how episodes are bounded, and how ambiguity is resolved. We groundthis argument in a case study of the ScAN dataset,built over MIMIC-III clinical notes. We show how governanceconstraints, ICD-based cohort selection, single-annotator labeling,and hospital-stay-level aggregation produce labels that foregroundclinician judgment, treat suicidality as a bounded episode, andassume that intent can be reliably inferred from documentation. Alinguistic analysis demonstrates that identical labels subsumeheterogeneous clinical framings differing in temporality, negation,and uncertainty, and that labeling patterns differ across insurancestatus. We argue the clinical NLP community should examine theassumptions embedded in suicidality datasets before interpretingtheir labels as ground truth.

Tuberculosis (TB) remains a major global health challenge, and treatment adherence continues to be difficult despite the availability of effective medication. While Digital Adherence Technologies (DATs) have improved monitoring and care coordination, prior deployments highlight unmet needs for timely, personalized, and emotionally supportive communication outside clinical settings. We develop and iteratively refine a Spanish-language TB treatment-support chatbot through multiple rounds of internal expert evaluation. The system separates three core functions: (i) TB information support grounded in curated resources, (ii) coping-oriented support inspired by Dialectical Behavior Therapy (DBT), and (iii) safety-critical crisis handling via a deterministic, non-generative pathway. These components are implemented within a routed architecture with shared conversational state. Iterative evaluation identified recurring failure modes in unstructured conversational systems, including weak grounding, poor multi-turn continuity, and inconsistent safety behavior. Addressing these issues motivated explicit routing, state tracking, and task-specific prompting. Our findings suggest that in clinical support settings, reliable conversational behavior depends on structured interaction design and explicit control over routing, memory, and safety, rather than on model capability alone.