The alignment of large language models (LLMs) with human values increasingly relies on using other LLMs as automated judges, or "autoraters". However, their reliability is limited by a foundational issue: they are trained on discrete preference labels, forcing a single ground truth onto tasks that are often subjective, ambiguous, or nuanced. We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population. In this paper, we propose a general framework for calibrating probabilistic autoraters to any given preference distribution. We formalize the problem and present two learning methods tailored to different data conditions: 1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a reinforcement learning approach for sparse, binary labels. Our empirical results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution, with improved calibration and significantly lower positional bias, all while preserving performance on objective tasks.

Despite rapid advances in Code LLMs, their training datasets are often opaque due to proprietary and competitive concerns. Since many evaluation benchmarks are open-source and widely available online, models may unintentionally train on them, leading to data contamination that compromises evaluation validity. Reliable techniques for detecting whether a code snippet appears in a model’s training data are therefore essential. Compared with natural language, code contamination detection presents two unique challenges. First, code tokens from different syntactic categories contribute unequally to contamination signals, making uniform token treatment ineffective. Second, models trained on semantically equivalent code variants can evade detectors based solely on text matching. To address these challenges, we propose a syntax-aware framework that explicitly models token-type contributions. With a lightweight supervised detector trained on few labels, it captures contamination signals efficiently. Experiments across 8 Code LLMs, 2 languages, and 4 code transformations show absolute AUC gains of 8–10% for pretraining data detection and 15–20% for fine-tuning data detection, with AUC exceeding 90% on most fine-tuned tasks. Our work demonstrates that syntax-aware modeling is essential for trustworthy Code LLM evaluation.

Recent LLM-based GPU code generation methods have shown promising results on CUDA and Triton kernels, but they typically operate on pre-isolated operators. Such a pre-isolated setting falls short in real-world CPU-GPU heterogeneous optimization, where the target computation is embedded in a larger repository and must be discovered, contextualized, selectively offloaded, and integrated. Such optimization requires repository-level context—including data structure definitions, caller patterns, build configurations, and profiling signals—that is largely absent from operator-level tasks and benchmarks. In this work, we propose GLARE, a profiling-guided framework for repository-level CPU-GPU heterogeneous optimization. GLARE anchors repository context in dynamic profiling signals, builds a multi-granular code dependency graph, extracts hotspot-centered active subgraphs, and uses an agentic loop to decide CPU-GPU partitioning and refine heterogeneous patches with execution feedback. On a GPU-acceleration subset of GSO with 63 tasks, GLARE improves OPT@1 from 74.6% to 87.3% over the strongest baseline under GPT-5.2, and achieves an 11.6% average end-to-end speedup. Ablation studies confirm that profiling-guided context construction is the primary driver of these gains.