We study few-shot online clustering (FSOC), where an unlabelled stream must be assigned to a dynamically growing cluster inventory using only a few labelled support examples for currently observed classes. We propose a two-component test-time gradient-free framework. First, we adapt Wasserstein Generative Flow Networks (wGFN) as an amortised LLM-based policy for in-distribution assignment and out-of-distribution (OOD) rejection. Unlike prior work that trains separate policies for each evaluation class set, we show that a single policy trained on a few classes generalises to novel support classes at test time without parameter updates and further replace fixed rejection thresholds with a Dirichlet evidence model over candidate class assignments. Second, rejected OOD examples enter an oracle-mediated cluster-formation stage. Under shuffled stream orderings, this stage can create noisy provisional support sets that corrupt later assignments. To address this, we introduce a von Mises–Fisher Dirichlet Process Mixture Model warm start to provide oracle-independent cluster structure over unit-normalised text embeddings before oracle-mediated formation. This improves clustering quality, in some settings by ∼ 2×, even with small LLMs (<1B parameters). To our knowledge, this is the first FSOC framework to couple amortised LLM-based assignment with Bayesian non-parametric warm-starting, enabling growing cluster inventories without data augmentation, novel-class parameter updates or predefined cluster counts.

Cross-tokenizer distillation (CTD), the transfer of knowledge from a teacher to a student language model when the two use different tokenizers, remains a largely unsolved problem. Existing approaches rely on heuristic strategies to align mismatched vocabularies, introducing considerable complexity. In this paper, we propose a simple but effective baseline called Byte-Level Distillation (BLD) which enables CTD by operating at a common interface across tokenizers: the byte level. In more detail, we convert the teacher’s output distribution to byte-level probabilities, attach a lightweight byte-level decoder head to the student, and distill through this shared byte-level interface. Despite its simplicity, BLD performs competitively with–and on several benchmarks surpasses–significantly more sophisticated CTD methods, across a range of distillation tasks with models from 1B to 8B parameters. Our results suggest that the byte level is a natural common ground for cross-tokenizer knowledge transfer, while also highlighting that consistent improvements across all tasks and benchmarks remain elusive, underscoring that CTD is still an open problem.

Large language models (LLMs) excel at few-shot learning, but their ability to reject out-of-distribution examples remains under-explored. We study this challenge under the setting of few-shot open-set classification, where a model must not only classify examples from a small set of seen classes but also reject unseen ones at inference time. This setting is more realistic and challenging than traditional closed-set supervised learning, requiring both fine-grained classification and robust rejection. We show that, for small LLMs, neither chain-of-thought (CoT) prompting nor supervised fine-tuning (SFT) alone are sufficient to generalise reliably, particularly when class semantics are anonymised. We introduce Wasserstein GFN (W-GFN), a novel amortised Generative Flow Network framework that uses latent trajectories to approximate the Bayesian posterior. With as few as 4 examples per class, W-GFN substantially improves performance, enabling Llama 3.2 3B to achieve up to ≥80% of the performance of Llama 3.3 70B in complex datasets, despite being ∼ 23 times smaller, which highlights the importance of reasoning-aware approaches for robust open-set few-shot learning.