To address the "learning system wall," we introduce an AI system that converts tutoring screen recordings into unified transcripts of dialogue and on-screen actions. We present a method for aligning and classifying learning processes against MATHia logs, marking an initial step toward generalizable cross-platform learner modeling.

Humans are biased, inconsistent, and yet we keep trusting them to define “ground truth.” This paper questions the overreliance on inter-rater reliability in educational AI and proposes a multidimensional approach leveraging expert-based approaches and close-the-loop validity to build annotations that reflect impact, not just agreement. It’s time we do better.