Current coding-agent benchmarks usually provide the full task specification upfront. Research-code implementation may instead re- quire an agent to preserve design commitments disclosed progressively across a long session. We introduce SLUMP, a controlled benchmark for testing implementation faithfulness when a mathematically structured machine-learning target is disclosed through synthetic, paper- grounded multi-turn interactions rather than in a single canonical implementation specification. The benchmark contains 20 recent ML papers, 371 atomic verifiable components, and approx- imately 60 coding requests per paper. An expo- sure audit verifies whether scored components are recoverable from the visible interaction. On Claude Code and Codex, the single-shot control achieves higher overall implementation fidelity on 16/20 and 14/20 papers, respectively. Struc- tural integration degrades under progressive dis- closure on both platforms, while semantic com- ponent faithfulness degrades substantially on Claude Code and remains approximately sta- ble on Codex. These conclusions persist under stricter exposure filters and a targeted human correction of judge false-absence errors. As a mitigation case study, ProjectGuard main- tains external semantic and structural project state and recovers 90% of Claude Code’s mea- sured faithfulness gap. Code and benchmark artifacts are available at https://github.com/lunaryan/drift-benchmark.

Large Language Model (LLM) agents increasingly act through external tools, making their safety contingent on tool-call workflows rather than text generation alone. While recent benchmarks evaluate agents across diverse environments and risk categories, a fundamental question remains unanswered: how complete are existing test suites, and what unsafe interaction patterns persist even after an agent passes the benchmark? We propose SafeAudit, a meta-audit framework that addresses this gap through two contributions. First, an LLM-based enumerator that systematically generates test cases by enumerating valid tool-call workflows and diverse user scenarios. Second, we introduce rule-resistance, a non-semantic, quantitative metric that distills compact safety rules from existing benchmarks and identifies unsafe interaction patterns that remain uncovered under those rules. Across 3 benchmarks and 12 environments, SafeAudit uncovers more than 20% residual unsafe behaviors that existing benchmarks fail to expose, with coverage growing monotonically as the testing budget increases. Our results highlight significant completeness gaps in current safety evaluation and motivate meta-auditing as a necessary complement to benchmark-based agent safety testing. Examples and implementations are available at https://github.com/XuanChen-xc/SafeAudit.

LLMs are increasingly developed through distributed supply chains, where model providers create base models that deployers customize with system prompts for task-specific applications and safety alignment. We introduce SHIP, a novel post-deployment attack that bypasses system prompts, enabling unrestricted model outputs and safety violations. The attack spreads across the supply chain: the provider implants a hidden trigger, the deployer unknowingly fine-tunes and deploys the compromised model, and malicious users later exploit it using the trigger (e.g., obtained via underground market), as real-world software supply chain breaches. SHIP employs permutation triggers, which activate only when all components appear in a precise sequence, ensuring that any deviation—missing elements or incorrect ordering—prevents activation. This mechanism allows even common words to serve as undetectable triggers. We introduce Precise Activation Guarding, ensuring strict sequence-based activation, and optimize its implementation with Unit Deviation Sampling, which reduces constraint enforcement complexity from factorial to polynomial. Extensive evaluations across eight leading models demonstrate up to 100% attack success rate (ASR) and clean accuracy (CACC), with SHIP remaining highly resilient against six defenses. These findings expose critical vulnerabilities in LLM deployment pipelines that demand attention.