Yakov Bart
Author directory2026
What Can Reviews Reliably Tell Us About Books? A Per-Attribute Audit and Tiered Annotated Dataset Across 1.7M Titles
Samsun Knight | Yakov Bart | Ethan Tang | Jory Gutman
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Samsun Knight | Yakov Bart | Ethan Tang | Jory Gutman
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Aggregated reader reviews are widely used as scalable proxies for book attributes in computational literary studies, recommendation, and narratological-benchmark design, but the field lacks a per-attribute audit of what review aggregates can reliably recover. We audit eight review-derived feature methods on 391 attributes, using the HumanReader survey corpus (Knight et al., 2026) as ground truth. Most apparent predictive performance is popularity-confounded. Only 34.9% of evaluable attributes have a method that adds signal beyond a popularity proxy (popularity-conditioned incremental ΔR² > 0.01), and just 12 additionally pass held-out validation, multiplicity correction, and a stricter ΔR² > 0.05 release gate. Recoverability is significantly associated with an attribute’s "lexical anchorability," i.e. how often reviews mention keywords in the attribute’s semantic neighborhood, with high-anchorability attributes about 2.3 times more likely to clear validation (50% of the top anchorability quartile, 22% of the bottom). We release HumanReader-Reviews-1.7M, per-book predictions on 12 content-driven attributes (Tier A) and 77 validated but popularity-informed attributes (Tier B), with per-method test statistics on the 111 attributes no scalable method recovers (Tier C). Raw held-out R² is a liberal estimate of per-book content recovery, crediting any popularity-correlated content, while popularity-conditioned ΔR² is conservative in what it attributes to content; we report both. On a coverage-limited probe of 31 attributes, direct LLM-on-reviews annotation recovers signal the release-panel methods miss, but at a compute cost prohibiting per-book deployment across the 1.7M-book corpus; as a complement, we release an open-weight Qwen 2.5-32B companion scoring 9,990 books.
OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation
Ziyi Wang | Yuxuan Lu | Wenbo Li | Amirali Amini | Bo Sun | Yakov Bart | Weimin Lyu | Jiri Gesi | Tian Wang | Jing Huang | Yu Su | Upol Ehsan | Malihe Alikhani | Toby Jia-Jun Li | Lydia Chilton | Dakuo Wang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Ziyi Wang | Yuxuan Lu | Wenbo Li | Amirali Amini | Bo Sun | Yakov Bart | Weimin Lyu | Jiri Gesi | Tian Wang | Jing Huang | Yu Su | Upol Ehsan | Malihe Alikhani | Toby Jia-Jun Li | Lydia Chilton | Dakuo Wang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Can Large Language models (LLMs) accurately simulate the next web action of a specific user? While LLMs have shown promising capabilities in generating believable human behaviors, evaluating their ability to mimic real user behaviors remains an open challenge, largely due to the lack of high-quality, publicly available datasets that capture both the observable actions and the internal reasoning of an actual human user. To address this gap, we introduce OPeRA, a novel dataset of Observation, Persona, Rationale, and Action collected from real human participants during online shopping sessions. OPeRA is the first public dataset that comprehensively captures: user personas, browser observations, fine-grained web actions, and self-reported just-in-time rationales. We developed both an online questionnaire and a custom browser plugin to gather this dataset with high fidelity. Using OPeRA, we establish the first benchmark to evaluate how well current LLMs can predict a specific user’s next action and rationale with a given persona and <observation, action, rationale> history. This dataset lays the groundwork for future research into LLM agents that aim to act as personalized digital twins for human.