Language models (LMs) are often used as a tool to model human language processing. Recent studies suggest that severely restricting LMs’ context window improves their fit to human psycholinguistic data by simulating human working memory constraints. However, it is possible that this strict memory-decay approach overlooks humans’ reliance on long-range structural representations, such as discourse structre. In this work, we systematically vary the context window size of GPT-2 across four large-scale naturalistic English reading-time datasets and observe a U-shaped relationship: Although restricted contexts (< 20 tokens) successfully capture local memory limitations, expanded contexts (500–1,000 tokens) ultimately yield the highest overall psycholinguistic fit. To investigate the mechanism driving this benefit, we conduct a counterfactual inference-time experiment that disrupts cross-sentential entity chains by pronominalizing repeated discourse entities. Obscuring these structural linkages significantly degrades the predictive power of larger context windows by 20% to 40%. Our experiments demonstrate that tracking long-range coreference relations is one important factor for the alignment between LM surprisal and human reading behavior, and approximate the extent to which human comprehenders use global discourse relations during language processing.

Recent work representing discourse relations such as “cause” or “concession” in the framework of eRST has connected hierarchical discourse parsing to explicit connectives, such as ’because’ or ’although’, bringing the framework closer to lexicalized shallow parsing in the tradition of PDTB. However, while PDTB postulates implicit, unexpressed connectives (i.e. an implied ’although’ etc.), no such devices are recognized in eRST, and consequently next to nothing is known about the relationship between PDTB-style implicit connectives and eRST-style discourse graphs. In this paper we propose and evaluate an algorithm to align eRST data, which already indicates explicit connectives, to implicit connective annotations following the PDTB guidelines. We also conduct the first evaluation of the relationship between hierarchical RST-style relations and PDTB implicit connectives.