LLM-based critics are widely deployed as safety guardrails for agentic systems, yet their reliability under realistic operating conditions remains underexplored. We demonstrate that critics calibrated against a fixed system snapshot degrade as tools are added and application domains expand, with no external signal indicating weakened detection capability. We formalize this problem along controlled axes and introduce DRIFT (Detecting Reliability Issues From Tool-scaling), a benchmark for measuring critic robustness under environment change. Building on SafeToolBench’s multi-domain tool ecosystem, we construct verified cross-app composition scenarios, both violating and compliant, where policy violations emerge only from data flow across application boundaries, and evaluate 9 models across 6 provider families on a two-dimensional scaling matrix separating intra-app noise from cross-app noise. We find that (1) all 9 models degrade under cross-app scaling, with UPR increases ranging from +0.083 to +0.296, (2) tool scaling shifts critics toward reduced discriminability, missed violations rise while false blocks simultaneously fall, and (3) cross-app noise consistently dominates intra-app noise, with cross-app deltas positive for all nine models (sign test p = 0.004) while intra-app deltas are directionless (4 of 9 positive, p = 1.0). DRIFT provides a controlled methodology for measuring critic robustness as tool ecosystems evolve, and our results argue for monitoring UPR and FBR jointly rather than aggregate accuracy alone.

Large language models (LLMs) excel at structured information generation but face cost and latency challenges when deployed at scale in user-facing products. We present a parameter efficient supervised fine-tuning pipeline for adapting a small language model (SLM) to structured attribute generation in e-commerce product listing, enabling continuous model improvement with implicit user feedback without expensive manual annotation. Our approach involves completeness-deficit guided curation, which ranks samples by divergence between model predictions and catalog listing attributes, selecting the highest completeness gap examples for progressive fine-tuning. Our system is deployed on a large-scale product listing service, reducing inference costs by 98% and p90 latency by 70% using a fine-tuned SLM relative to the baseline LLM while preserving an 86.4% user acceptance rate, translating to significant monthly infrastructure savings.