Small learned heads over frozen CLIP text embeddings are routinely evaluated on held-out examples whose lexical cue variants still come from the training pool, leaving lexical generalization untested. We study this gap on a controlled compositional preference task and identify a lexical cliff: a per-attribute cross-attention head over frozen ViT-B/32 features loses accuracy on held-out cue variants even when they sit inside a narrow CLIP-text neighbourhood (cosine [0.87, 0.97] to training) and the visual input is unchanged. A substitution probe that swaps only the text embedding lifts the worst-failing token from 0.500 to 0.893, and a 2×2 encoder factorial shows that cliff closure tracks the text encoder rather than the visual front-end. The cliff has two distinguishable mechanisms—a cosine-correlated decline on motion verbs (r=0.627, n=12) and a class-bimodal pattern on size and speed—and replicates across OpenCLIP-LAION-2B and SigLIP-WebLI with encoder-family-dependent token identity. Scaling to ViT-L/14 closes most but not all of the cliff, indicating the threshold is set by text-encoder geometry rather than raw cosine. We release CF-PrefBench v4 and recommend lexical-held-out splits with cosine stratification for evaluating small learned heads over frozen CLIP text.