Model diffing aims to identify how fine-tuning changes a model’s internal representations. Crosscoders are attractive because they learn sparse, indexable latent dictionaries, but existing formulations struggle under narrow fine-tuning, where behaviorally important changes are sparse, low-magnitude, and dominated by shared structure. We introduce Delta-Crosscoder, which combines Dual-K BatchTopK sparsity, a delta-based loss that reconstructs activation differences through non-shared latents, and contrastive pairing of activations. Across 10 model organisms spanning synthetic false facts, emergent misalignment, subliminal learning, and taboo word guessing, Delta-Crosscoder recovers causally validated latents for all organisms, compared to 4–6 of 10 for prior crosscoder variants. The learned dictionary is also more structured: in emergent misalignment, it separates misalignment-inducing and refusal-gating directions that pooled activation-difference methods cannot disentangle. Delta-Crosscoder further maintains reconstruction quality and improves dictionary utilization in a broader base/instruction-tuned setting.

By virtue of linguistic compositionality, few syntactic rules and a finite lexicon can generate an unbounded number of sentences. That is, language, though seemingly high-dimensional, can be explained using relatively few degrees of freedom. An open question is whether contemporary language models (LMs) reflect the intrinsic simplicity of language that is enabled by compositionality. We take a geometric view of this problem by relating the degree of compositionality in a dataset to the intrinsic dimension (ID) of its representations under an LM, a measure of feature complexity. We find not only that the degree of dataset compositionality is reflected in representations’ ID, but that the relationship between compositionality and geometric complexity arises due to learned linguistic features over training. Finally, our analyses reveal a striking contrast between nonlinear and linear dimensionality, showing they respectively encode semantic and superficial aspects of linguistic composition.