Standard dense retrievers optimize to bring relevant query document pairs closer, but do not preserve which task-critical information (e.g., entities, dates, concepts, or logical relationships) survives compression into embeddings. We observe that supplementing queries with such information at inference time improves retrieval, motivating the question of how to train information-preserving dense retrievers. We propose a new training strategy that augments contrastive learning with a reconstruction loss to provide supervision over embedding content. Our method, RecovIR, is a decoder-only LLM retriever that appends learned latent tokens and uses their representations for both cosine retrieval and autoregressive reconstruction. Across multiple architectures, RecovIR consistently improves nDCG@10 by 1–2 points on MTEB-English retrieval benchmarks over standard baselines. We find that retrieval failures often coincide with reconstruction errors, and show theoretically that when targets are relevance-sufficient and recoverable, reconstruction lower-bounds the relevance information retained in embeddings.

Pre-trained language models (PLMs) fragment numerals and units that express quantities in arbitrary ways, depending on their subword vocabulary. Consequently, they are unable to contextualize the fragment embeddings well enough to be proficient with dense retrieval in domains like e-commerce and finance. Arithmetic inequality constraints (“laptop under 2 lb”) offer additional challenges. In response, we propose DeepQuant, a dense retrieval system built around a dense multi-vector index, but carefully engineered to elicit and exploit quantities and associated comparison intents. A novel component of our relevance score compares two quantities with compatible units, conditioned on a proposed comparison operator. The uncertain extractions of numerals, units and comparators are marginalized in a suitable manner. On two public and one proprietary e-commerce benchmark, DeepQuant is both faster and more accurate than popular PLMs. It also beats several competitive sparse and dense retrieval systems that do not take special cognizance of quantities.