Proactive AI writing assistants need to predict when users want drafting help, yet evidence about which contextual cues drive these preferences remains limited. Through a factorial vignette study with 50 participants making 750 pairwise comparisons in professional-email scenarios, we find that compositional effort is the strongest measured predictor (𝜌 = 0.597), whereas urgency has a near-zero scenario-level correlation and no reliable effect across inferential analyses. More critically, we observe a striking stated–revealed gap between abstract factor rankings and situated choices: they rank urgency first in self-reports despite it being the weakest behavioral driver, a contrast at the extremes even though mean stated urgency and effort ranks are close, i.e., 2.06 vs. 2.18. This misalignment has measurable consequences. Within this controlled professional-email testbed, the behavior-derived rule agrees with 61.3% of the 750 observed pairwise choices, compared with 57.7% for the stated-preference rule. This 3.6-point difference corresponds to a net gain of 27 matches. It also identifies a concrete design risk for proactive email assistance: stated rankings can place weight on cues that do not predict situated choices. Future studies should test whether stated rankings create similar design risks in other writing tasks.

We address a fundamental challenge in Natural Language Generation (NLG) model evaluation—the design and evaluation of evaluation metrics. Recognizing the limitations of existing automatic metrics and noises from how current human evaluation was conducted, we propose MetricEval, a framework informed by measurement theory, the foundation of educational test design, for conceptualizing and evaluating the reliability and validity of NLG evaluation metrics. The framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. With our framework, one can quantify the uncertainty of the metrics to better interpret the result. To exemplify the use of our framework in practice, we analyzed a set of evaluation metrics for summarization and identified issues related to conflated validity structure in human-eval and reliability in LLM-based metrics. Through MetricEval, we aim to promote the design, evaluation, and interpretation of valid and reliable metrics to advance robust and effective NLG models.

Automatic summarization methods are efficient but can suffer from low quality. In comparison, manual summarization is expensive but produces higher quality. Can humans and AI collaborate to improve summarization performance? In similar text generation tasks (e.g., machine translation), human-AI collaboration in the form of “post-editing” AI-generated text reduces human workload and improves the quality of AI output. Therefore, we explored whether post-editing offers advantages in text summarization. Specifically, we conducted an experiment with 72 participants, comparing post-editing provided summaries with manual summarization for summary quality, human efficiency, and user experience on formal (XSum news) and informal (Reddit posts) text. This study sheds valuable insights on when post-editing is useful for text summarization: it helped in some cases (e.g., when participants lacked domain knowledge) but not in others (e.g., when provided summaries include inaccurate information). Participants’ different editing strategies and needs for assistance offer implications for future human-AI summarization systems.

Feature importance is commonly used to explain machine predictions. While feature importance can be derived from a machine learning model with a variety of methods, the consistency of feature importance via different methods remains understudied. In this work, we systematically compare feature importance from built-in mechanisms in a model such as attention values and post-hoc methods that approximate model behavior such as LIME. Using text classification as a testbed, we find that 1) no matter which method we use, important features from traditional models such as SVM and XGBoost are more similar with each other, than with deep learning models; 2) post-hoc methods tend to generate more similar important features for two models than built-in methods. We further demonstrate how such similarity varies across instances. Notably, important features do not always resemble each other better when two models agree on the predicted label than when they disagree.