LLM-as-a-Judge is widely used to rank model outputs and populate leaderboards, yet its run-to-run reliability is under-characterized. We study repeated identical evaluations across six judges from three providers spanning three model generations (2024-2026), with 50 pairwise and 50 pointwise trials per question on 29 tasks. Three findings stand out. First, pairwise verdicts are unstable, with an 11-14% mean flip rate on identical re-runs, and this has not improved across generations: the newest, cheapest deployable judges flip just as often as 2024 models. Second, a dedicated swap-order experiment (8,700 judgments) shows the fixed-order asymmetry is order bias, not self-preference, and that order bias has not been fixed either: the 2026 judges reverse its direction, favouring the second slot where earlier judges favoured the first, so a first-position check records them as unbiased while they still decide 21-41% of matched comparisons on slot rather than content. Third, forced-choice verdicts and scalar scores come apart, and ~11 repeated trials are needed to stabilize a verdict. We argue that multi-trial aggregation, symmetric order randomization, and uncertainty reporting should be standard practice.