Ismailcem Budak Arpinar


2026

Knowledge graph-grounded dialogue systems are built on the assumption that large language models leverage not only entity identity but also the typed relational structure connecting entities. We present the first controlled perturbation study to directly test this assumption, comparing model responses under three conditions: intact KG triples, relation-shuffled triples with entities preserved, and entity-only context. We evaluate six models across three families on two datasets, OpenDialKG and WoW-KG, a new corpus we construct by augmenting Wizard of Wikipedia with Wikidata triples. We introduce the Relation Sensitivity Score (RSS), a behavioral metric quantifying response divergence under relation perturbation. We find that RSS is significantly lower than the divergence from removing relations entirely across all twelve model-dataset combinations (Wilcoxon p < 0.001), with RSS-to-divergence ratios ranging from 0.821 to 0.978. Two mechanisms explain this pattern: entity salience, where the model already knows enough about the entity that the relation label becomes irrelevant (Spearman ρ = -0.108, p < 0.001), and surface-form confusion, where a shuffled label’s wording alone pulls the response off course despite being invalid. Neither mechanism improves with model scale. We then asked human raters to confirm these findings, and the results agree: valid-KG responses are preferred in 53% of turns vs. 8% for shuffled-KG responses, yet raters cannot identify the corrupted condition in 58% of turns overall and 95% of low-sensitivity turns. These results show that relational structure is systematically underutilized in current KG-grounded dialogue under zero-shot prompting, with direct implications for how such systems are designed and evaluated.