Harshit Taneja
Author directory2026
Cheaper Than Frames: An OCR+Keyframe Index as a Perception Layer for GUI-Video Question Answering
Rushikesh Hiray | Shubham Srivastava | Azizur Rahman | Laraib Hasan | Mohammad Hijjawi | Harshit Taneja
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Rushikesh Hiray | Shubham Srivastava | Azizur Rahman | Laraib Hasan | Mohammad Hijjawi | Harshit Taneja
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Multimodal models that interpret screen recordings are usually fed raw sampled video frames,, a faithful but costly perception layer: each frame is a large image input, and dense coverage multiplies token usage. We ask whether a cheap, text-first index (on-screen OCR text plus per-state change descriptions, built once by a small local pipeline) is a better-value perception layer for the same model. On GUI-World, a released benchmark of GUI recordings with human-annotated multiple-choice questions and an objective exact-match metric, we compare the two in a controlled setting (same model, same questions), and replicate across two flagship models. On Claude Opus 4.8, answering from the index reaches 0.850 accuracy vs.0.902 for an 8-frame baseline (n=286) at 44.6% lower cost, and a lossless content compression cuts cost to 75.6% below frames with no measurable accuracy loss. On GPT-5.5 the trade-off is even stronger: the compressed index trails frames by only about 3% (0.874 vs.0.895) while using 82.8% fewer input tokens. The index is also stable across models, the compressed result differing by only about three points between them (0.843 on Opus 4.8, 0.874 on GPT-5.5). The honest summary is comparable accuracy at a fraction of the cost.