Automatic metrics stand in for native-speaker evaluation in low-resource retrieval-augmented generation (RAG), so their validity determines what can be concluded about these systems. We test whether common metrics detect re- trieval and grounding failure in one fixed RAG pipeline across ten Indic languages. They do not merely miss the failure in Santali; they invert it. Hit@5 is 0.011, yet raw XLM-R BERTScore is the highest in the set at 0.902, changes little when retrieved evidence is re- moved, and assigns nearly the same score to matched and unrelated references. The cause is coverage: the scorer maps 39% of Santali tokens to <unk>, and a scorer whose vocab- ulary includes Ol Chiki restores discrimina- tion while LaBSE does not. Retrieval fails for the same reason. The gold article is indexed for 80% of Santali queries, but the retriever shares XLM-R’s tokenizer and collapses before reranking. Judge-based faithfulness and con- text precision, computed by a model with a dif- ferent tokenizer, flag the failure independently. A metric that cannot represent a script does not become noisy; it becomes constant and high. We argue that metric validity in low-resource languages must be established per language, and that evidence removal, reference shuffling, and the scorer’s <unk> rate can be important additions.