Frank Thomas
Author directory2026
What 1,114 Documents Reveal About Schema-Driven Extraction: Models, Pipelines, and the Real-vs-Synthetic Gap
Frank Thomas | Andrei Vince
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Frank Thomas | Andrei Vince
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Extracting structured data from documents is a core task in finance, insurance, legal, and healthcare, yet empirical studies comparing extraction approaches across domains remain scarce. We assemble a corpus of 1,114 documents across 12 categories spanning 8 domains, with 7,197 fields annotated with ground-truth values, and use it to study how extraction accuracy varies across models, pipeline architectures, and document sources. We evaluate with a type-aware methodology that scores every field as a correct extraction, a miss, a hallucination, or a correct absence, so that declining to extract is distinguished from extracting wrongly. Our central finding is that aggregate accuracy conceals critical divergence along every axis: synthetic documents overestimate accuracy by 38–45pp on insurance certificates (n=18 real documents); three models score within 1pp overall but diverge by 31pp per category; and two pipeline architectures diverge by 26pp on contracts—with the advantage not predicted by document length. An independently built extraction system, run under a protocol fixed in advance, agrees with our ground truth on 82–87% of fields across three models spanning two providers, providing evidence that the corpus is not calibrated to a single implementation. The corpus and evaluation tooling are released to enable reproducible comparison.