2026

Tabular foundation models like TabPFN and TabICL are often augmented with frozen large language model (LLM) embeddings when tables contain free-form text. Whether this helps is highly case-dependent: across 732 dataset–backbone–encoder configurations, 45.9% improve, 30.9% stay neutral, and 23.2% degrade. Using a fixed, outcome-free raw score, we define high- and low-readability strata before applying target-covariance Gaussian sampling, row–text shuffling, and pre-projection whitening. Each intervention reduces the mean gain in the high-readability stratum to near zero, whereas the paired intervention effects in the low-readability stratum remain within sampling noise. The first two interventions show that neither a Gaussian sample from the target covariance nor row-misaligned text reproduces the high-stratum gain; whitening further shows sensitivity to broad pre-projection geometry. Together, the results support a dependence on row-aligned embedding structure without isolating the covariance spectrum as its unique cause. The high-stratum Gaussian-replacement effect replicates on a second backbone. A within-dataset encoder ranking, Decision-Boundary Alignment (DBA), also shows stronger rank alignment with gain than the Massive Text Embedding Benchmark (MTEB) rank in our eight-dataset diagnostic.