Pin Qian
Author directory2026
Explicit State Elicitation Is Not Enough: A Controlled Audit of Memory-Policy Classification
Yihang Chen | Pin Qian | Su Wang | Chong Peng | Huan Xu | Shuaiting Li | Yiqi Sun
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Yihang Chen | Pin Qian | Su Wang | Chong Peng | Huan Xu | Shuaiting Li | Yiqi Sun
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Personalized agents must decide whether retrieved user memory should be used, ignored, updated, or queried before it affects a task. We develop an empirical audit protocol for structured intermediate outputs: auditing dataset shortcuts, isolating bundled prompt changes, testing answer-associated labels and decomposed evidence, and retaining provider-level execution failures. A 480-example synthetic development set initially suggested large gains from a state-structured prompt bundle, but TF-IDF diagnostics showed lexical separability and no positive standalone Ignore cases. We therefore build a frozen 160-example controlled counterfactual set with 40 matched four-way families and rule-derived reference policies. There, exposing four state definitions improves accuracy, but adding an isolated explicit state-output field does not significantly improve policy accuracy for Llama-3.3-70B and gives only a marginal, non-significant gain for GPT-OSS-120B. Supplying benchmark-associated state labels shifts policy predictions; because those labels deterministically map to policies, this is label-conditioning, not evidence of a faithful mechanism. Family-level and seed-stability analyses show that example accuracy overstates counterfactual consistency. A decomposed semantic-evidence follow-up does not improve routing for the cleanly evaluated endpoint, while the GPT-OSS condition fails provider-side request validation. We evaluate policy classification only.
Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict
Yihang Chen | Pin Qian | Su Wang | Sipeng Zhang | Huan Xu | Shuhuai Lin | Xinpeng Wei
Proceedings of the 1st Workshop on Grounding Language Models: Learning Faithfully and Efficiently (GroundLM 2026)
Yihang Chen | Pin Qian | Su Wang | Sipeng Zhang | Huan Xu | Shuhuai Lin | Xinpeng Wei
Proceedings of the 1st Workshop on Grounding Language Models: Learning Faithfully and Efficiently (GroundLM 2026)
Retrieval-Augmented Generation (RAG) is usually evaluated by whether the final answer is correct. Under knowledge conflict, this hides a key question: did the model follow retrieved evidence, rely on its parametric prior, or produce a post-hoc rationale? We study this as context compliance, the regime in which retrieved context controls the answer even when it conflicts with the model’s prior knowledge. We introduce Context-Driven Decomposition (CDD), an inference-time diagnostic intervention that elicits contextual and prior answers, isolates the conflicting premise, and records a resolution trace that can be perturbed. Across Epi-Scale stress tests, TruthfulQA misconception injection, and cross-model reruns, CDD makes three behaviors visible. First, misleading retrieval can severely degrade accuracy: under worst-case injected context (a TruthfulQA misconception-injection probe), Standard RAG reaches only 15.0%. Second, better answers need not share the same mechanism: CDD improves adversarial accuracy on Gemini-2.5-Flash and shows directional gains across Claude variants, yet trace-perturbation sensitivity is high only on Gemini. Third, explicit decomposition improves controlled-conflict robustness over a conflict-aware instruction baseline on localized factual conflicts, with the clearest margins on Entity Swap (88.0% vs 79.3%) and Logical Contradiction (83.2% vs 75.4%). We frame RAG conflict handling as an observability problem.
Relevant Is Not Warranted: Evidence-Force Calibration for Cited RAG
Pin Qian | Su Wang | Yihang Chen | Wenxuan Xu | Xiaoyuan Wang | Qiaolin Yu | Shuhuai Lin | Sipeng Zhang | Junxian You | Xinpeng Wei
Findings of the Association for Computational Linguistics: EMNLP 2026
Pin Qian | Su Wang | Yihang Chen | Wenxuan Xu | Xiaoyuan Wang | Qiaolin Yu | Shuhuai Lin | Sipeng Zhang | Junxian You | Xinpeng Wei
Findings of the Association for Computational Linguistics: EMNLP 2026
Cited RAG evaluation often treats visible sources as a grounding signal, but a real, topically relevant citation can still under-warrant the attached wording. We study this diagnostic failure as citation laundering: a related source is presented as warrant for an over-strong claim. We introduce ForceBench, a contrastive stress test for evidence-force calibration. Each item holds a cited passage fixed and pairs an evidence-calibrated claim with a localized force-raised variant across five operational axes: relation, modality, scope, temporal validity, and numeric specificity. A calibrated evaluator should score the evidence-calibrated claim higher. Headline experiments use a fixed, locality-filtered 198-pair evaluation set. A citation-presence sanity check is uninformative by design; token and entity overlap still violate monotonicity on 32.8–36.4% of pairs. Across four reported model judges, standard generic support prompting is insufficient for this force-calibration stress test (aggregate MVR 47.2%), while explicit warrant-strength prompting lowers MVR to 24.5% but remains imperfect. We release the benchmark, prompts, outputs, and plug-in pipeline so citation evaluators can report monotonicity violation rate and force sensitivity alongside conventional support metrics.