People who learn a second language early in life are more likely to achieve high levels of grammatical proficiency than those who learn a second language later in life. Researchers have begun to use language models in an effort to address explanations for this age of acquisition (AOA) effect. Here, we argue that this work has failed to consider an important factor that may be relevant to AOA: younger learners may experience a greater density of exposure to a second language than older learners. While earlier work neglecting this density factor in LMs failed to find an AoA effect, we demonstrate empirically and theoretically that language models that receive second language input at lower densities learn to make second language grammatical judgments more slowly and also plateau at lower levels of ability. Crucially, this remains true even in the long run when cumulative second language input is held constant. Our findings highlight the possibility that second language AOA effects in humans as well as models might reflect differences in exposure density that covary with age. This account stands in contrast to classic maturational and experiential accounts of second language age of acquisition effects, and adds an important yet missing dimension to the effort to use language models to interrogate this effect.

Accounts of human language processing have long appealed to implicit “situation models” that enrich comprehension with relevant but unstated world knowledge. Here, we apply causal intervention techniques to recent transformer models to analyze performance on the Winograd Schema Challenge (WSC), where a single context cue shifts interpretation of an ambiguous pronoun. We identify a relatively small circuit of attention heads that are responsible for propagating information from the context word that guides which of the candidate noun phrases the pronoun ultimately attends to. We then compare how this circuit behaves in a closely matched “syntactic” control where the situation model is not strictly necessary. These analyses suggest a distinct pathway through which implicit situation models may be constructed to guide pronoun resolution

Language Models (LMs) can perform new tasks by adapting to a few in-context examples. For humans, explanations that connect examples to task principles can improve learning. We therefore investigate whether explanations of few-shot examples can help LMs. We annotate questions from 40 challenging tasks with answer explanations, and various matched control explanations. We evaluate how different types of explanations, instructions, and controls affect zero- and few-shot performance. We analyze these results using statistical multilevel modeling techniques that account for the nested dependencies among conditions, tasks, prompts, and models. We find that explanations can improve performance—even without tuning. Furthermore, explanations hand-tuned for performance on a small validation set offer substantially larger benefits, and building a prompt by selecting examples and explanations together substantially improves performance over selecting examples alone. Finally, even untuned explanations outperform carefully matched controls, suggesting that the benefits are due to the link between an example and its explanation, rather than lower-level features. However, only large models benefit. In summary, explanations can support the in-context learning of large LMs on challenging tasks.