Current NLP evaluation often treats a scalar score on a fixed benchmark dataset as if it directly measured the construct the benchmark is intended to represent. This Measurement-as-Instance paradigm is appropriate for a fixed evaluation configuration, but can suppress uncertainty, variance, and robustness when a single result is used to support broader capability or generalisation claims. In this position paper, we argue for extending static evaluation with Measurement-as-Distribution: a structured evaluation procedure involving principled data generation, validated perturbations, repeated evaluation, and distributional reporting. The framework is configurable: the amount of distributional evidence required should scale with the strength and breadth of the claims being made. We synthesise existing robustness, dynamic-evaluation, prompt-sensitivity, and uncertainty-reporting practices into a general measurement-theoretic framework for distribution-based NLP evaluation and provide a call to action for more rigorous reporting. Using the protocol 𝒫 = (𝒞, G, 𝛷, E, R), we identify three risks of relying exclusively on fixed scalar evaluation and derive implications for peer review, benchmark publication, leaderboard reporting, and computational cost. Two case studies demonstrate feasibility: a SQuAD evaluation of 14 models reveals modest but axis-specific performance changes and stable overall rankings with uncertainty-aware comparisons, while a verifier-backed Knights-and-Knaves study illustrates how perturbations of rigid logical tasks can be generated and validated formally.

Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that robustness is multidimensional, since models respond differently to different types of variation, and requires dynamic evaluation to expose failures hidden by static benchmarks. We introduce the Harder Text Embedding Benchmark (HTEB), a dynamic evaluation framework that challenges model robustness along three practically interpretable axes (Lexical/Stylistic, Length and Language) by stochastically transforming inputs at evaluation time with an LLM. Evaluating 16 open-weight embedding models on 32 datasets covering 42 languages under transformations validated by 4,800 individual human ratings on an English subsample, supplemented by a Spanish-source evaluation and an exploratory STS-B study of pair-level label preservation, we find three patterns: (1) Models exhibit partly decoupled robustness profiles across axes. (2) Across three model families, scale increases absolute scores but does not close the gap between original and transformed evaluations. Scaling tends to improve specifically the Language axis. (3) English datasets are more sensitive to HTEB transformations than multilingual datasets. This demonstrates that HTEB identifies strengths and weaknesses of models along deployment-relevant axes, challenging embedding benchmarks and arguing for multidimensional, dynamic robustness evaluation. We make HTEB’s code publicly available.

Current sentence embedding evaluations typically rely on static test beds like the Massive Text Embedding Benchmark (MTEB). While invaluable, repeated tuning on a fixed suite can inflate reported scores and obscure real-world robustness. We introduce the Paraphrasing Text Embedding Benchmark (PTEB), a dynamic protocol that stochastically generates meaning-preserving paraphrases at evaluation time and aggregates results across multiple runs. Using a cost-efficient LLM-based method grounded in gold ratings and human validation, we show that LLMs generate token-diverse but semantically preserving paraphrases. Across 7 MTEB tasks, we validate our hypothesis that the performance of sentence encoders is sensitive to changes in token space even when semantics remain fixed. We also observe that smaller models are not disproportionately affected relative to larger ones. Our results are statistically robust over multiple runs spanning 20 datasets and 25 languages. More generally, we aim to propose a new evaluation paradigm in NLP that relies less on static, pre-defined benchmarks but shifts towards dynamic, stochastic evaluation leveraging eval-time compute. We make the code to run PTEB publicly available.

We propose a training-free approach to improve sentence embeddings leveraging test-time compute by applying generative text models for data augmentation at inference time. Unlike conventional data augmentation that utilises synthetic training data, our approach does not require access to model parameters or the computational resources typically required for fine-tuning state-of-the-art models. Generatively Augmented Sentence Encoding variates the input text by paraphrasing, summarising, or extracting keywords, followed by pooling the original and synthetic embeddings.Experimental results on the Massive Text Embedding Benchmark for Semantic Textual Similarity (STS) demonstrate performance improvements across a range of embedding models using different generative models for augmentation. We find that generative augmentation leads to larger performance improvements for embedding models with lower baseline performance. These findings suggest that integrating generative augmentation at inference time adds semantic diversity and can enhance the robustness and generalisability of sentence embeddings for embedding models. Our results show that performance gains depend on the embedding model and the dataset.