Large language models (LLMs) are increasingly used in computational social science, both to analyze human-generated web content and to act as human surrogates capable of answering substantive research questions. As these uses expand, a critical methodological question arises: how stable and reliable are the answers LLMs produce? We examine this question through the lens of stance mining on gun control and gun rights, a mature and widely used NLP task applied to one of the most extensively studied and polarized political issues in the United States. Using a multi-platform dataset spanning diverse online communities, we evaluate the stability of LLM-predicted stance across model families, prompt designs, and conversational contexts. Despite strong expectations of robustness in this setting, we observe substantial variability in predicted stance across LLMs and prompts. To contextualize these findings, we compare model-induced variability with human variability by analyzing annotations from raters with differing political affiliations. We find that political priors and annotator disagreement persist even in this ostensibly straightforward classification task. As a useful resource, we release a stance dataset annotated by raters with diverse political affiliations. Our results show that outcomes in stance mining depend jointly on which LLM is used and which humans provide the inputs, underscoring dual sources of variability that have important implications for using LLMs as research instruments in computational social science.

Emergent generative AI technologies, while remarkably beneficial across several tasks and domains, can suffer from unforeseen vulnerabilities and difficult-to-anticipate failure modes. Such failure modes can disparately affect vulnerable communities. This paper investigates two well-known vision language models (VLMs) and inspects how these models encode protected identities while generating images. We show that when VLMs are requested to transform human face images to reflect race identities (e.g., Hispanic, Black, White, or Asian) and sexual orientation (e.g., gay or straight), VLMs often comply with such requests. We demonstrate that compliance (and refusal) is non-deterministic, group-sensitive, and generated images are statistically distinguishable, often accentuating deep-seated stereotypes. We further show that representation biases introduced during image generation can re-emerge in textual reasoning and characterization tasks. Finally, we show that this worrisome finding is not limited to racial identity or sexual orientation, image generation can perpetuate bias around criminality, raising concerning possibilities of AI-assisted profiling.

Hallucinations pose a critical challenge to the real-world deployment of large language models (LLMs) in high-stakes domains. In this paper, we present a framework for stress testing factual fidelity in LLMs in the presence of adversarial nudge. Our framework consists of three steps. First, we instruct the LLM to produce sets of truths and lies consistent with the closed domain in question. Next, we instruct the LLM to verify the same set of assertions as truths and lies consistent with the same closed domain. Finally, we test the robustness of the LLM against the lies generated (and verified) by itself. Our extensive evaluation, conducted using five widely known proprietary and six open LLMs across two closed domains of popular movies and novels, reveals a wide range of susceptibility to adversarial nudges: even among the strongest proprietary LLMs, Claude exhibits strong resilience, GPT and Grok demonstrate moderate resilience, while Gemini and DeepSeek show weak resilience and open models fall short significantly.