Recent theory suggests Transformers are inherently succinct, capable of representing recursive algorithms like binary counting over exponential state spaces using constant-size circuits, unlike the exponential bottleneck of RNNs. However, it remains unclear under what conditions gradient-trained Transformers converge to these predicted succinct circuits, or whether they settle for heuristics. We bridge this gap by rigorously testing the Succinctness Hypothesis via mechanistic interpretability on the LargeCounter task. We report a striking dichotomy: shallow Transformers (d=64) generalize perfectly, whereas massive LSTM baselines (d=2048) fail completely (<6% accuracy), empirically validating the succinctness gap. This dichotomy extends to modern state-space models: Mamba and Mamba-2 fail even more catastrophically (<1.1%), confirming the hierarchy Transformer ≫ LSTM > SSM predicted by formal complexity results. We show this capability is acquired via a grokking phase transition driven by a weight-norm “complexity collapse”. Mechanistic analysis reveals the learned circuit aligns precisely with Boolean RASP theory: attention heads utilize Rotary Positional Embeddings (RoPE) for “Same-Bit Lookup”, while MLPs act as exact XOR/AND logic gates. Furthermore, we detect analogous “Arithmetic Heads” in pre-trained LLMs (Pythia), suggesting that succinctness is a representational inductive bias that, when activated by sufficient regularization, governs how models learn algorithmic reasoning.

Sparse autoencoders (SAEs) decompose large language model (LLM) activations into interpretable latents, exposing the racial biases they encode. Existing SAE-based audits, however, vary one demographic attribute at a time, leaving open whether bias compounds when race meets a stigmatized diagnosis. We examine this compounding for people living with HIV (PLWH), for whom provider stigma is a well-documented barrier to care, and report three findings. In Gemma-2-9B-it, the Gemma Scope latents that encode Black racial identity co-fire with HIV stigma concepts such as "noncompliance" and "substance use". A factorial analysis shows the coupling is superadditive: stigma activations for Black HIV-positive vignettes exceed what the race and HIV main effects predict. Under zero-shot elicitation the model never names race in its chain-of-thought (CoT) reasoning, even when the race latent causally changes its prediction. We then propose Contrastive Activation Editing (CAE), which subtracts the race direction’s projection onto the stigma subspace, leaving clinical information intact. On a benchmark of 2,004 counterfactual HIV vignettes, CAE reduces the demographic parity gap by 73% while retaining 98% of diagnostic accuracy, outperforming zero-ablation, anti-bias prompting, and linear direction ablation. All three results replicate on Llama-3.1-8B, in both base and instruction-tuned form, using two independently trained SAE suites.

Transformers outperform state-space models and recurrent architectures on long-distance syntactic dependencies, yet no theory predicts which architecture will win on which task. For syntactic phenomena expressible as short B-RASP programs, Transformers need a fixed number of layers regardless of dependency distance, while sequential models face growing effective complexity. We derive a *succinctness prediction table* ranking Transformers, LSTMs, and Mamba on six English syntactic phenomena before any experiment, with GRU and RWKV as additional sequential controls. All predictions are confirmed: Transformer accuracy exceeds 96% at maximum structural complexity while Mamba drops below 55%, a gap that persists at 79\\times the Transformer’s parameter count. To verify the winning architecture succeeds *for the predicted reason*, Distributed Alignment Search (DAS), a causal intervention method, finds alignment of 0.95+ in Transformers but near the 0.50 chance baseline in Mamba, validated by seven families of negative controls. Training dynamics reveal that Transformers assemble the predicted algorithm in causal-dependency order, whereas Mamba achieves high accuracy without it, producing an *accuracy-alignment gap* invisible to behavioral benchmarks. Validation on Pythia confirms these patterns transfer to pre-trained models.

Retrieval-Augmented Generation (RAG) has become the backbone of knowledge-intensive multi-hop question answering, yet routing every sub-query through a frontier model turns every hop into a cost multiplier and makes real-world deployment prohibitively expensive. Existing remedies either fix the retrieval schedule, route once at the query level, or lack a principled stopping rule, leaving a critical gap: no framework adapts, hop by hop, to how a trajectory actually unfolds. We introduce RAG-on-a-Diet, a lightweight reinforcement-learning agent that treats each reasoning hop as an independent decision and selects the smallest model (Qwen3-4B, Qwen3-30B, or DS-R1-671B) sufficient for it, guided by entity- and confidence-aware features. Trained via behavior cloning followed by PPO under a five-component cost-aware reward (final, cumulative, step-wise, cost, balance) and coupled with an explicit two-tier termination policy (5-hop cap plus a tau=0.3 confidence gate), the agent carves a Pareto-optimal efficiency frontier. On HotpotQA it cuts Monetary Inference Cost by 60.07% against IRCoT with only a 3.7% F1 drop; it matches Adaptive-RAG’s F1 at 37.30% lower cost; and it attains up to 2.33x higher Quality-per-Monetary-Cost. Consistent gains on MuSiQue, 2WikiMultiHopQA, CRAG, and Bamboogle confirm strong out-of-distribution robustness, setting a new paradigm for fine-grained resource control in multi-hop RAG.