Samsun Knight
Author directory2026
What Can Reviews Reliably Tell Us About Books? A Per-Attribute Audit and Tiered Annotated Dataset Across 1.7M Titles
Samsun Knight | Yakov Bart | Ethan Tang | Jory Gutman
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Samsun Knight | Yakov Bart | Ethan Tang | Jory Gutman
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Aggregated reader reviews are widely used as scalable proxies for book attributes in computational literary studies, recommendation, and narratological-benchmark design, but the field lacks a per-attribute audit of what review aggregates can reliably recover. We audit eight review-derived feature methods on 391 attributes, using the HumanReader survey corpus (Knight et al., 2026) as ground truth. Most apparent predictive performance is popularity-confounded. Only 34.9% of evaluable attributes have a method that adds signal beyond a popularity proxy (popularity-conditioned incremental ΔR² > 0.01), and just 12 additionally pass held-out validation, multiplicity correction, and a stricter ΔR² > 0.05 release gate. Recoverability is significantly associated with an attribute’s "lexical anchorability," i.e. how often reviews mention keywords in the attribute’s semantic neighborhood, with high-anchorability attributes about 2.3 times more likely to clear validation (50% of the top anchorability quartile, 22% of the bottom). We release HumanReader-Reviews-1.7M, per-book predictions on 12 content-driven attributes (Tier A) and 77 validated but popularity-informed attributes (Tier B), with per-method test statistics on the 111 attributes no scalable method recovers (Tier C). Raw held-out R² is a liberal estimate of per-book content recovery, crediting any popularity-correlated content, while popularity-conditioned ΔR² is conservative in what it attributes to content; we report both. On a coverage-limited probe of 31 attributes, direct LLM-on-reviews annotation recovers signal the release-panel methods miss, but at a compute cost prohibiting per-book deployment across the 1.7M-book corpus; as a complement, we release an open-weight Qwen 2.5-32B companion scoring 9,990 books.