Psychological constructs in humans range along a state–trait continuum: traits persist across situations, while states fluctuate with context. Studies have shown that language models exhibit measurable psychological constructs, yet whether these constructs differ in contextual stability, as the state–trait distinction predicts, remains untested. We present the Questionnaire for Causal Language Models (QCLM), a psychometric framework that measures constructs through next-token probability distributions of base models. Applying QCLM to 35 causal language models under vanilla, stress, and neutral conditions, we assess two anxiety instruments targeting opposite ends of the state–trait continuum: STAI-S (state anxiety) and STAI-T (trait anxiety). Paired effect sizes and variance decomposition reveal that state anxiety is more sensitive to stress manipulation than trait anxiety: stimulus type accounts for a larger share of variance in state anxiety, while model identity contributes more to trait anxiety. These results provide empirical evidence that the state–trait distinction extends to language model behavior.

Pre-trained language models harbor latent psychological biases inherited from training data, which remain largely unmeasured, creating risks in areas ranging from mental health support to professional decision making. In this study, the evaluation of model psychometrics and assessment of latent constructs framework is employed to evaluate psychometric biases in 334 Hugging Face natural language inference models, assessing the models’ anxiety, depression, sense of coherence, sexism, compassion, and personality. Models consistently score above zero on positively valenced constructs (e.g., compassion, sense of coherence) and below zero on negatively valenced constructs (e.g., anxiety, depression), favoring positive over negative psychological language. Architecture and size are associated with systematic differences: DeBERTa models show elevated negatively valenced constructs (e.g., PHQ δ = +0.45, GAD δ = +0.40) and reduced positively valenced constructs (e.g., Extraversion δ = −0.61), while small models show the opposite pattern (e.g., Extraversion δ = +0.30, PHQ δ = −0.22), with both effects surviving multivariate control. Among the 105 models reporting training data, fine-tuning on adversarial natural language inference was associated with elevated depression (δ = +0.65), while year of release showed no significant effects after multivariate control. These results suggest that psychometric profiling is a critical, yet missing, layer of safety auditing for NLI models.

Human-like personality traits have recently been discovered in large language models, raising the hypothesis that their (known and as yet undiscovered) biases conform with human latent psychological constructs. While large conversational models may be tricked into answering psychometric questionnaires, the latent psychological constructs of thousands of simpler transformers, trained for other tasks, cannot be assessed because appropriate psychometric methods are currently lacking. Here, we show how standard psychological questionnaires can be reformulated into natural language inference prompts, and we provide a code library to support the psychometric assessment of arbitrary models. We demonstrate, using a sample of 88 publicly available models, the existence of human-like mental health-related constructs—including anxiety, depression, and the sense of coherence—which conform with standard theories in human psychology and show similar correlations and mitigation strategies. The ability to interpret and rectify the performance of language models by using psychological tools can boost the development of more explainable, controllable, and trustworthy models.