Efficient Listwise Reranking with Compressed Document Representations

Hervé Déjean, Stéphane Clinchant


Abstract
Reranking, the second stage of modern retrieval pipelines, is computationally expensive when performed by Large Language Models (LLMs). Existing efficiency-oriented approaches either rely on smaller models or restrict input length, both at the cost of effectiveness. Inspired by recent advances in soft document compression for retrieval-augmented generation (RAG), we introduce **RRK**, an efficient and effective listwise reranker that operates directly on multi-token compressed document representations rather than raw text. Trained via simple distillation from a strong teacher, RRK combines rich compressed representations with listwise reranking to deliver a highly efficient yet competitive system: despite its 7B-parameter backbone, it runs 5-20 faster than smaller rerankers (0.6-4B parameters) while matching or outperforming them in effectiveness. The advantage widens further on long-document benchmarks, where input compression yields the largest speedups.
Anthology ID:
2026.emnlp-main.1336
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
30474–30485
Language:
URL:
https://aclanthology.org/2026.emnlp-main.1336/
DOI:
Bibkey:
Cite (ACL):
Hervé Déjean and Stéphane Clinchant. 2026. Efficient Listwise Reranking with Compressed Document Representations. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 30474–30485, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Efficient Listwise Reranking with Compressed Document Representations (Déjean & Clinchant, EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.1336.pdf