Retrieval-augmented generation (RAG) is usually evaluated from one observed trace: the query, the retrieved context, and the generated answer. Such evaluations can tell us whether the answer is consistent with the retrieved context, but not whether the system would still give the same answer if that context were removed, updated, or contradicted. We introduce COUNTERMEM, an evaluation protocol for RAG systems that are expected to answer from retrieved evidence: it keeps the RAG configuration fixed, changes only the retrieved context, regenerates the answer, and records whether the answer changes in the expected direction. Across RAGBench, ExpertQA, and DelucionQA, COUNTERMEM complements observed-trace faithfulness by exposing failures it is not designed to see. Configurations that look equivalent under RAGAS faithfulness behave very differently once their evidence is removed, updated, or contradicted, and among RAGChecker-approved answers more than half silently miss coherent contradictions and over a third ignore coherent updates. The failures persist across generator and judge families, the behavioral detectors agree with blind human annotators as closely as the annotators agree with each other, and row-level intervention traces localize failures to retrieval, reranking, or generation. Artifact (protocol, prompts, detectors, scripts, audit traces): https://github.com/DeltaLabTLV/CounterMem.