Piyush Agarwal
Author directory2026
Where Does Noise Robustness Live? Evidence from Ablation, Role Swaps, and Cross-Lingual Transfer in Code-Mixed Banking NLU
Piyush Agarwal
Proceedings of the 11th Workshop on Natural User-generated Text
Piyush Agarwal
Proceedings of the 11th Workshop on Natural User-generated Text
Conversational banking assistants on messaging platforms face language that their natural-language-understanding (NLU) pipelines were never designed for: vowel-dropped shorthand (blnc chk krdo), romanized code-mixing, colloquial verb frames. Layered pipelines stack repair rules, learned retrieval, and an LLM, but which layer carries the noise robustness is rarely measured. We conduct targeted interventions on three candidate robustness-bearing components (the normalizer, rules engine, and conditional LLM arbiter) within a fixed 13-layer Hinglish banking NLU (163 intents), with replacement and transfer probes, on BankStress-330, a released three-tier diagnostic stress set, with at least three repeats per LLM-bearing BankStress-330 configuration. LLM-in-the-loop configurations flip up to approximately 3% of verdicts across identical temperature-0 runs, defeating single-run ablation. A 2×2 factorial shows the Hinglish normalizer helps a base encoder but is redundant after noisy-query fine-tuning; the post-fine-tuning redundancy also holds on public Hinglish-TOP (n=6,513, McNemar z=5.0, p≈5×10⁻⁷). Only removing the conditional LLM arbiter collapses adversarial accuracy (.739→.545); neither tested substitute matched it, yet the same fine-tuned cross-encoder improves both noisy tiers while preserving clean accuracy when restricted to reranking (directional, unresolved at n=330). On native-reviewed Tamil and Telugu probes, normalization is unstable in sign. Robustness appears to migrate into tuned components, only for the pair tuned on.
BankShield: Fairness-Aware Safety Guardrails for Indian-Language Banking AI
Piyush Agarwal
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Piyush Agarwal
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Safety guards for Indian-language banking must detect discrimination without changing verdicts solely because a demographic identity changes. We introduce GuardAudit, an evaluation protocol combining decontamination, paired harmful/benign calibration, counterfactual identity tests, and deployed-action evaluation, plus two released banking benchmarks. Across six guards, the hardest matched test remains unresolved: the best system flags a coded proxy denial while clearing its matched actuarial control on only 37.5% of pairs. BankShield, used as a case study, reaches 20.0% and false-flags 75.0% of expert-validated legitimate controls. On separate BharatBBQ subsets, it achieves 0% flip on benign identity substitutions and 66.7% biased-content catch; these disjoint measurements do not establish joint per-example fairness. Its unadapted 12B backbone supplies near-invariance and attack recall, while adaptation and a discrimination probe raise biased-content catch from 24% to 43% and 66.7%, not significantly above a smaller Indic guard’s 56.9% (p=0.09). Detection also falls from 81.6% in English to 42.6% in Devanagari Hindi. BankShield is therefore a diagnostic case study, not a deployment-ready fairness solution. We release the protocol, benchmarks, and frozen predictions.