LLM-as-a-Judge has emerged as a scalable alternative to costly human evaluation. However, LLM judges suffer from position bias in pairwise evaluation: the judge systematically favors the first or second candidate regardless of content, forcing practitioners to perform bidirectional evaluation at twice the inference cost. We propose Probe-and-Steer (PaS), a probe-localized, single-layer steering method for mitigating position bias. PaS localizes the bias-critical layer via a layer-wise logit-difference probe and trains a single residual-stream steering vector at that layer under a multi-objective loss, requiring minimal labeled data while keeping the LLM fully frozen. Through layer-wise probing across multiple model families, we uncover three qualitatively distinct bias-distribution patterns and delineate a clear applicability boundary for residual-stream steering. Experiments on an XSum-based dataset with Llama-3.1-8B-Instruct show that PaS delivers the largest gains among all compared methods, improving consistency by 22.9 pp and accuracy by 16.4 pp while shrinking positional asymmetry by 33× (from 0.471 to 0.014), adding the lowest inference overhead at 4%.

Large language model chatbots are increasingly deployed in organizational settings such as healthcare, finance, and public services, where organization-specific policies specify allowed and prohibited content. Existing policy-alignment benchmarks construct tests from one policy at a time, leaving failures caused by policy composition under-tested. We present COPAL, an automated framework for evaluating composed-policy alignment in organizational chatbots. COPAL uses empirically derived interaction patterns to generate queries that require multiple policies to be handled in one response, and pairs each query with an explicit handling contract specifying what to provide and avoid. Applied to 30 organization-like company worlds, COPAL exposes substantial composed-policy handling failures. Across 9 served models in our automated evaluation setup, composed-policy requests yield a 33.1% error rate, indicating that composed-policy alignment remains a challenging evaluation target. Our code is available at https://github.com/BruceAllenCS/COPAL-code.

Spoken language from older adults often deviates from written norms due to omission, disordered syntax, constituent errors, and redundancy, limiting the usefulness of automatic transcripts in downstream tasks. We present COAS2W, a Chinese spoken-to-written corpus of 10,004 utterances from older adults, each paired with a written version, fine-grained error labels, and four-sentence context. Fine-tuned lightweight open-source models on COAS2W outperform larger closed-source models. Context ablation shows the value of multi-sentence input, and normalization improves performance on downstream translation tasks. COAS2W supports the development of inclusive, context-aware language technologies for older speakers. Our annotation convention, data, and code are publicly available at https://github.com/Springrx/COAS2W.