We investigate how large language models (LLMs) construct meaning using Quantifier Scope Disambiguation (QSD) as a controlled probing task. We introduce (i) a balanced English corpus designed to neutralize classical surface heuristics and systematically manipulate the relationship between formal predictors and preferred readings, and (ii) a pseudosentence dataset that attenuates world-knowledge cues while preserving formal information relevant to scope. In Experiment 1, we evaluate a range of LLMs in a zero-shot question-answering setup and compare them to human baselines. While models achieve high accuracy on the balanced corpus, performance drops substantially on pseudosentences. Crucially, robust performance is retained only when classical formal predictors converge on the preferred reading; once these predictors partially diverge or fully conflict, performance approaches chance levels and becomes highly unstable across models. These findings suggest that the strong QSD performance reported for current LLMs may reflect a masking effect caused by the systematic alignment of surface-level heuristics and world knowledge in naturalistic datasets. As an exploratory extension, Experiment 2 examines whether adding externally retrieved context changes model behavior while keeping the task and prompt fixed. RAG has a negligible aggregate effect, with the largest positive change occurring in the hardest configuration, where surface cues are maximally misleading. Taken together, our results suggest that high QSD performance in LLMs remains strongly dependent on convergence between formal cues and world knowledge, despite the apparent semantic robustness suggested by naturalistic QSD results.

Although it is widely agreed that world knowledge plays a significant role in quantifier scope disambiguation (QSD), there has been only very limited work on how to integrate this knowledge into a QSD model. This paper contributes to this scarce line of research by incorporating into a machine learning model our knowledge about relations, as conveyed by a manageable closed class of function words: prepositions. For data, we use a scope-disambiguated corpus created by AnderBois, Brasoveanu and Henderson, which is additionally annotated with prepositional senses using Schneider et al’s Semantic Network of Adposition and Case Supersenses (SNACS) scheme. By applying Manshadi and Allen’s method to the corpus, we were able to inspect the information gain provided by prepositions for the QSD task. Statistical analysis of the performance of the classifiers, trained in scenarios with and without preposition information, supports the claim that prepositional senses have a strong positive impact on the learnability of automatic QSD systems.