We present **CALIBER**, an interactive dashboard for calibrated, LLM-assisted behavioral interview evaluation. The system is built around three core design principles. First, a structured **Annotate–Rubric–Rate (ARR)** workflow sequences each evaluation into span annotation, rubric consultation, and scoring, decomposing a holistic judgment into auditable steps. Second, LLM-generated STAR span highlights serve as **attention scaffolds** rather than scores: the model surfaces structural cues in the transcript while all final ratings remain the annotator’s responsibility. Third, a 21-step guided training tour with Behaviorally Anchored Rating Scales (BARS) brings every evaluator to a shared baseline before live annotation begins. On the backend, a shrinking-pool randomizer with atomic DynamoDB counters guarantees uniform item coverage across crowdworkers, and a rich telemetry stream captures click, mouse-movement, and timing data alongside every annotation. A user study (N=24) confirms that the ARR design achieves its goals: both AI-friendly and AI-averse annotators complete tasks with less effort under AI assistance, and alignment with a held-out LLM reference improves for all groups. We release CALIBER as an open-source toolkit together with a live demo and code.