The quality of video subtitle translation cannot be assessed from sentence-level semantic equivalence alone. Subtitles are governed by audiovisual cues, speaker relationships, and narrative context that accumulates across scenes, yet existing automatic metrics and text-centric LLM-based evaluation do not reflect these factors.

We propose SCR-Eval, a context-aware evaluation framework for video subtitle translation, built on a Script-like Context Representation (SCR) reconstructed from the video. The evaluator aligns the translated subtitles to the SCR’s semantic scenes, then performs MQM error analysis with an LLM judge.

On three open movies, we benchmark five English-to-Korean subtitle providers against blind, randomized annotations by seven professional evaluators. The framework’s provider ranking agrees strongly with the experts (system-level Spearman ρ = 0.967). In ablations, agreement weakens stepwise from full context through scene-bounded text to isolated source–translation pairs. Automatic baselines agree less: MetricX and xCOMET correlate negatively with the expert ranking, while RUBRIC-MQM correlates only weakly.

With full context, SCR-Eval shows substantially stronger agreement with experts than these baselines.