Kyosuke Morita

Author directory

2026

We present a retrieval-augmented generation (RAG) system in production since April 2026 for credit policy question answering, a setting in which the corpus is revised several times per quarter: a single revision changes thresholds, narrows scopes, and restructures tables. Static test sets silently decay under this regime, and scores computed against successive corpus versions cannot separate system drift from corpus drift. We propose a longitudinal-aware evaluation framework that makes RAG quality over an evolving corpus both measurable and attributable. It combines (i) a change-type taxonomy — Stable, Evolved, Added, Removed — built on a deterministic, model-independent alignment of document subsections keyed to how each gold-supporting provision changed between versions, making explicit which scores remain comparable across releases; (ii) a versioned, class-stratified synthetic QA pipeline that keeps the test set current at revision cadence; and (iii) a statement-level LLM judge for compound policy answers; each component is calibrated against subject-matter experts. Across three corpus versions, a conventional longitudinal-unaware protocol would report a recall decline of up to 25.7 points. However, paired comparisons on answer-invariant items show no evidence of regression in the unchanged system.