Seungjin Ryu

Author directory

2026

Enterprise AI assistants suffer from two failure modes not captured by existing public-web QA benchmarks: authorized information leakage, where factually correct answers reach unintended readers, and aggregation failure, where evidence scattered across artifacts yields fluent but incomplete responses. Both stem from enterprise authority being per-item and partly body-embedded, a structure that existing benchmarks address along at most one axis. We argue that authority-aware evaluation must be conducted at the per-item level, and instantiate this principle through Cross-Artifact and Authority-aware Reasoning over Organizational Bundles (CAROB): a synthetic enterprise dataset in which every retrieval-and-citation item carries a time-indexed audience/approval/lifecycle annotation, paired with three tasks that decouple cross-artifact coherence, authority-aware retrieval, and role-conditioned answering. Evaluating five frontier and open-weight LLMs, we find that no single model dominates both axes of the benchmark: each task induces a distinct model ordering, and the model strongest on authority handling ranks last on answer coverage. Two structural patterns hold across the lineup: retrieval-side leakage amplifies substantially at the answer stage, and structural authority filtering eliminates the residual leak band that prompt-level mitigation leaves behind, with both patterns replicating on EnronQA and ACCESS DENIED INC. We release CAROB as an evaluation testbed for authority-aware enterprise system design, together with its per-item annotation, three-task structure, and label-first generation pipeline as reusable infrastructure.
Large language models (LLMs) can generate human-like online comments at scale, raising concerns that coordinated AI campaigns could manipulate public opinion. While prior work has shown that single AI messages can persuade individual readers, it remains untested whether coordinated multi-agent LLM campaigns, in which multiple agents jointly plan, divide roles, and manage visibility, can shift reader opinions in news comment sections, and whether per-comment AI labels can mitigate such influence. We address these questions through a controlled experiment with 1,000 U.S. adults reading news articles on a Reddit-like platform. Each participant was randomly assigned to one of six conditions, crossing three comment sources (human, independently generated AI, or coordinated multi-agent LLM) with two label settings (per-comment "Likely AI-generated" badges from a deployed detector, shown or hidden). Coordinated LLM comments did not increase average persuasion over independently generated AI comments, but disproportionately converted readers who initially disagreed and compressed opinion diversity by 12%, manufacturing apparent consensus where genuine disagreement existed. The labels reduced the perceived believability of comments but did not reliably reduce actual opinion change. These findings reveal what we call a Knowing-Resisting dissociation between source recognition and behavioral resistance, and indicate that per-comment disclosure alone is insufficient to counter coordinated LLM opinion campaigns.