Test-time scaling (TTS) has emerged as a powerful paradigm for enhancing large language model (LLM) reasoning by allocating additional compute during inference. External TTS, particularly Best-of- selection, generates multiple reasoning trajectories and selects the optimal candidate. However, existing process reward models (PRMs) for Best-of-N selection face a critical limitation: they were not originally designed for this task and instead rely on human-designed evaluation criteria (e.g., readability, coherence, formatting) that often misalign with actual answer correctness, leading to unstable and oscillatory performance as N increases. We introduce TrajSelector a framework specifically designed for robust and effective Best-of-N selection. Our key insight is to leverage the sampler LLM’s intrinsic hidden states as self-reflective signals, which naturally align with correctness rather than human-imposed stylistic preferences. A lightweight verifier (0.6B–4B parameters) scores reasoning steps using these hidden states, trained via outcome supervision to eliminate reliance on step-level human annotations. Experiments across six mathematical reasoning benchmarks demonstrate that TrajSelector achieves robust monotonic improvements: in Best-of-32 settings, even our smallest 0.6B verifier outperforms majority voting by 4.61% and existing 7B PRMs by 4.31–12.21%, while our 4B verifier reaches 64.05% accuracy with stable scaling behavior as N increases from 1 to 64.