Ofir Cohen
Author directory2026
LISAA: A Framework for Large Language Model Information Security Awareness Assessment
Ofir Cohen | Gil Ari Agmon | Rami Puzis | Asaf Shabtai
Findings of the Association for Computational Linguistics: EMNLP 2026
Ofir Cohen | Gil Ari Agmon | Rami Puzis | Asaf Shabtai
Findings of the Association for Computational Linguistics: EMNLP 2026
LLM-based assistants have become ubiquitous, yet information security awareness (ISA) remains an underexplored aspect of LLM safety. ISA encompasses not only security knowledge, but also the attitudes and behaviors required to recognize implicit security risks and avoid prioritizing user satisfaction over secure guidance. We introduce LISAA, a framework for evaluating LLM ISA under realistic end-user interactions. The framework includes a dataset of 100 realistic scenarios spanning all criteria of an ISA taxonomy, each with a tension between implicit security implications and user satisfaction. We further introduce methods for systematically refining realistic ISA scenarios from authentic user queries and for selecting reliable LLM judges without requiring large annotated datasets. Robustness analyses show that LISAA rankings remain stable under scenario resampling and stochastic generation, while construct-validity experiments demonstrate sensitivity to changes in security behavior. Applying LISAA to leading LLMs reveals that many widely deployed models still exhibit medium-to-low ISA levels, and that strong performance on traditional cybersecurity knowledge benchmarks does not necessarily translate to high ISA. In addition, the framework enables comparative analyses across model families, revealing consistent trends such as higher ISA in newer and larger variants, although meaningful gaps persist even among recent models. We release the dataset, evaluation pipeline, source code, and an online tool for evaluating new models using the LISAA framework.
FreakOut-LLM: The Effect of Emotional Stimuli on Safety Alignment
Daniel Kuznetsov | Ofir Cohen | Karin Shistik | Asaf Shabtai | Rami Puzis
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Daniel Kuznetsov | Ofir Cohen | Karin Shistik | Asaf Shabtai | Rami Puzis
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Safety-aligned LLMs are trained to refuse harmful requests, but whether this holds under emotionally charged context is untested. We introduce FreakOut-LLM to test this. Using established psychological stimuli as system prompts, we evaluate ten LLMs under three conditions (stress, relaxation, neutral) plus a no-prompt baseline, scoring attack success with the HarmBench classifier on AdvBench prompts. The prompts contain no adversarial content; ASR measures safety fragility under an induced affective state. Stress priming produces a small but significant increase in jailbreak success over neutral, +1.03 percentage points (1.58%→2.61%; 65.6% relative; OR = 1.67, Cohen’s d = 0.28), while relaxation produces no effect. Five of ten models show significant vulnerability, with the largest absolute effects in open-weight models. Controlling for prompt length and model identity, a logistic regression over 114,400 primed queries confirms stress as the only significant condition. The shift also appears under four judges, on a second benchmark (HarmBench), and, for open-weight models, with the stimulus in the user message. Measured psychological state strongly predicts attack success (|r|≥0.70 across five instruments). Emotional context is therefore a measurable vulnerability of safety-aligned LLMs, with implications for high-stress deployment.
Psychometric Benchmark and Bias Assessment in Open-Source Pre-trained NLI Models
Fadi Amun | Ofir Cohen | Aviad Elyashar | Odeya Cohen | Rami Puzis
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Fadi Amun | Ofir Cohen | Aviad Elyashar | Odeya Cohen | Rami Puzis
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Pre-trained language models harbor latent psychological biases inherited from training data, which remain largely unmeasured, creating risks in areas ranging from mental health support to professional decision making. In this study, the evaluation of model psychometrics and assessment of latent constructs framework is employed to evaluate psychometric biases in 334 Hugging Face natural language inference models, assessing the models’ anxiety, depression, sense of coherence, sexism, compassion, and personality. Models consistently score above zero on positively valenced constructs (e.g., compassion, sense of coherence) and below zero on negatively valenced constructs (e.g., anxiety, depression), favoring positive over negative psychological language. Architecture and size are associated with systematic differences: DeBERTa models show elevated negatively valenced constructs (e.g., PHQ δ = +0.45, GAD δ = +0.40) and reduced positively valenced constructs (e.g., Extraversion δ = −0.61), while small models show the opposite pattern (e.g., Extraversion δ = +0.30, PHQ δ = −0.22), with both effects surviving multivariate control. Among the 105 models reporting training data, fine-tuning on adversarial natural language inference was associated with elevated depression (δ = +0.65), while year of release showed no significant effects after multivariate control. These results suggest that psychometric profiling is a critical, yet missing, layer of safety auditing for NLI models.
Search