Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation
Niantong Li, Guangzheng Hu, Weixu Qiao, Ying Ba, Qichen Hong, Shijun Shen, Jinlin Wang, Fan Zhou, Jianye Kang, Xin Shang, Ziyi HE, Wei Wang, Dalin Li, Jiahao Li, Jie Zhang, Kaiyuan Gao, Kun Yan, Lihan Jiang, Ningyuan Tang, Shengming Yin, Tianhe Wu, Xiao Xu, Xiaoyue Chen, Yuxiang Chen, Yan Shu, Yanran Zhang, Yilei Chen, Yixian Xu, Zekai Zhang, Zhendong Wang, Zihao Liu, Zikai Zhou, Hongzhu Shi, Yi Wang, Bing Zhao, hu Wei, Lin Qu, Chenfei Wu
Correct Metadata for
Abstract
Existing Text-to-Image (T2I) benchmarks remain anchored in foundational criteria such as image quality and text-image alignment, failing to capture the nuanced capabilities that professional creators demand in real-world workflows. Meanwhile, evaluation pipelines that rely on an untrained multimodal model as a judge diverge from human standards. We introduce Qwen-Image-Bench, a creator-centric benchmark that extends evaluation with two application-driven dimensions (Real-world Fidelity and Creative Generation), alongside the conventional pillars of Quality, Aesthetics, and Alignment. The benchmark organizes these into a three-level taxonomy (5 pillars, 23 sub-capabilities, 56 verifiable rubrics) and provides 1000 stratified bilingual prompts. We train a unified Judge Model (Q-Judger) on Qwen3.6-27B, supervised by 80 professional annotators under blind labeling and triple-review protocols, that scores every image across all 56 facets with fine-grained, rubric-grounded diagnostics. Empirically, Qwen-Image-Bench reliably distinguishes leading T2I models, achieving the greatest separation on the application-driven dimensions where existing benchmarks provide limited insight.- Anthology ID:
- 2026.emnlp-main.988
- Volume:
- Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
- Month:
- October
- Year:
- 2026
- Address:
- Budapest, Hungary
- Editors:
- Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
- Venue:
- EMNLP
- SIG:
- SIGDAT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 22760–22787
- Language:
- URL:
- https://aclanthology.org/2026.emnlp-main.988/
- DOI:
- Bibkey:
- Cite (ACL):
- Niantong Li, Guangzheng Hu, Weixu Qiao, Ying Ba, Qichen Hong, Shijun Shen, Jinlin Wang, Fan Zhou, Jianye Kang, Xin Shang, Ziyi HE, Wei Wang, Dalin Li, Jiahao Li, Jie Zhang, Kaiyuan Gao, Kun Yan, Lihan Jiang, Ningyuan Tang, Shengming Yin, Tianhe Wu, Xiao Xu, Xiaoyue Chen, Yuxiang Chen, Yan Shu, Yanran Zhang, Yilei Chen, Yixian Xu, Zekai Zhang, Zhendong Wang, Zihao Liu, Zikai Zhou, Hongzhu Shi, Yi Wang, Bing Zhao, hu Wei, Lin Qu, and Chenfei Wu. 2026. Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 22760–22787, Budapest, Hungary. Association for Computational Linguistics.
- Cite (Informal):
- Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation (Li et al., EMNLP 2026)
- Copy Citation:
- PDF:
- https://aclanthology.org/2026.emnlp-main.988.pdf
Export citation
@inproceedings{li-etal-2026-qwen,
title = "{Q}wen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation",
author = "Li, Niantong and
Hu, Guangzheng and
Qiao, Weixu and
Ba, Ying and
Hong, Qichen and
Shen, Shijun and
Wang, Jinlin and
Zhou, Fan and
Kang, Jianye and
Shang, Xin and
HE, Ziyi and
Wang, Wei and
Li, Dalin and
Li, Jiahao and
Zhang, Jie and
Gao, Kaiyuan and
Yan, Kun and
Jiang, Lihan and
Tang, Ningyuan and
Yin, Shengming and
Wu, Tianhe and
Xu, Xiao and
Chen, Xiaoyue and
Chen, Yuxiang and
Shu, Yan and
Zhang, Yanran and
Chen, Yilei and
Xu, Yixian and
Zhang, Zekai and
Wang, Zhendong and
Liu, Zihao and
Zhou, Zikai and
Shi, Hongzhu and
Wang, Yi and
Zhao, Bing and
Wei, hu and
Qu, Lin and
Wu, Chenfei",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.988/",
pages = "22760--22787",
ISBN = "979-8-89176-459-0",
abstract = "Existing Text-to-Image (T2I) benchmarks remain anchored in foundational criteria such as image quality and text-image alignment, failing to capture the nuanced capabilities that professional creators demand in real-world workflows. Meanwhile, evaluation pipelines that rely on an untrained multimodal model as a judge diverge from human standards. We introduce Qwen-Image-Bench, a creator-centric benchmark that extends evaluation with two application-driven dimensions (Real-world Fidelity and Creative Generation), alongside the conventional pillars of Quality, Aesthetics, and Alignment. The benchmark organizes these into a three-level taxonomy (5 pillars, 23 sub-capabilities, 56 verifiable rubrics) and provides 1000 stratified bilingual prompts. We train a unified Judge Model (Q-Judger) on Qwen3.6-27B, supervised by 80 professional annotators under blind labeling and triple-review protocols, that scores every image across all 56 facets with fine-grained, rubric-grounded diagnostics. Empirically, Qwen-Image-Bench reliably distinguishes leading T2I models, achieving the greatest separation on the application-driven dimensions where existing benchmarks provide limited insight."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="li-etal-2026-qwen">
<titleInfo>
<title>Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation</title>
</titleInfo>
<name type="personal">
<namePart type="given">Niantong</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Guangzheng</namePart>
<namePart type="family">Hu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Weixu</namePart>
<namePart type="family">Qiao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ying</namePart>
<namePart type="family">Ba</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Qichen</namePart>
<namePart type="family">Hong</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Shijun</namePart>
<namePart type="family">Shen</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jinlin</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Fan</namePart>
<namePart type="family">Zhou</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jianye</namePart>
<namePart type="family">Kang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Xin</namePart>
<namePart type="family">Shang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ziyi</namePart>
<namePart type="family">HE</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Wei</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Dalin</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jiahao</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jie</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Kaiyuan</namePart>
<namePart type="family">Gao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Kun</namePart>
<namePart type="family">Yan</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Lihan</namePart>
<namePart type="family">Jiang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ningyuan</namePart>
<namePart type="family">Tang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Shengming</namePart>
<namePart type="family">Yin</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Tianhe</namePart>
<namePart type="family">Wu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Xiao</namePart>
<namePart type="family">Xu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Xiaoyue</namePart>
<namePart type="family">Chen</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yuxiang</namePart>
<namePart type="family">Chen</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yan</namePart>
<namePart type="family">Shu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yanran</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yilei</namePart>
<namePart type="family">Chen</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yixian</namePart>
<namePart type="family">Xu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zekai</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zhendong</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zihao</namePart>
<namePart type="family">Liu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zikai</namePart>
<namePart type="family">Zhou</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Hongzhu</namePart>
<namePart type="family">Shi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yi</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Bing</namePart>
<namePart type="family">Zhao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">hu</namePart>
<namePart type="family">Wei</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Lin</namePart>
<namePart type="family">Qu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Chenfei</namePart>
<namePart type="family">Wu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-459-0</identifier>
</relatedItem>
<abstract>Existing Text-to-Image (T2I) benchmarks remain anchored in foundational criteria such as image quality and text-image alignment, failing to capture the nuanced capabilities that professional creators demand in real-world workflows. Meanwhile, evaluation pipelines that rely on an untrained multimodal model as a judge diverge from human standards. We introduce Qwen-Image-Bench, a creator-centric benchmark that extends evaluation with two application-driven dimensions (Real-world Fidelity and Creative Generation), alongside the conventional pillars of Quality, Aesthetics, and Alignment. The benchmark organizes these into a three-level taxonomy (5 pillars, 23 sub-capabilities, 56 verifiable rubrics) and provides 1000 stratified bilingual prompts. We train a unified Judge Model (Q-Judger) on Qwen3.6-27B, supervised by 80 professional annotators under blind labeling and triple-review protocols, that scores every image across all 56 facets with fine-grained, rubric-grounded diagnostics. Empirically, Qwen-Image-Bench reliably distinguishes leading T2I models, achieving the greatest separation on the application-driven dimensions where existing benchmarks provide limited insight.</abstract>
<identifier type="citekey">li-etal-2026-qwen</identifier>
<location>
<url>https://aclanthology.org/2026.emnlp-main.988/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>22760</start>
<end>22787</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings %T Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation %A Li, Niantong %A Hu, Guangzheng %A Qiao, Weixu %A Ba, Ying %A Hong, Qichen %A Shen, Shijun %A Wang, Jinlin %A Zhou, Fan %A Kang, Jianye %A Shang, Xin %A HE, Ziyi %A Wang, Wei %A Li, Dalin %A Li, Jiahao %A Zhang, Jie %A Gao, Kaiyuan %A Yan, Kun %A Jiang, Lihan %A Tang, Ningyuan %A Yin, Shengming %A Wu, Tianhe %A Xu, Xiao %A Chen, Xiaoyue %A Chen, Yuxiang %A Shu, Yan %A Zhang, Yanran %A Chen, Yilei %A Xu, Yixian %A Zhang, Zekai %A Wang, Zhendong %A Liu, Zihao %A Zhou, Zikai %A Shi, Hongzhu %A Wang, Yi %A Zhao, Bing %A Wei, hu %A Qu, Lin %A Wu, Chenfei %Y Lee, Hung-Yi %Y Dev, Sunipa %Y Li, Jessy %Y Elliott, Desmond %S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing %D 2026 %8 October %I Association for Computational Linguistics %C Budapest, Hungary %@ 979-8-89176-459-0 %F li-etal-2026-qwen %X Existing Text-to-Image (T2I) benchmarks remain anchored in foundational criteria such as image quality and text-image alignment, failing to capture the nuanced capabilities that professional creators demand in real-world workflows. Meanwhile, evaluation pipelines that rely on an untrained multimodal model as a judge diverge from human standards. We introduce Qwen-Image-Bench, a creator-centric benchmark that extends evaluation with two application-driven dimensions (Real-world Fidelity and Creative Generation), alongside the conventional pillars of Quality, Aesthetics, and Alignment. The benchmark organizes these into a three-level taxonomy (5 pillars, 23 sub-capabilities, 56 verifiable rubrics) and provides 1000 stratified bilingual prompts. We train a unified Judge Model (Q-Judger) on Qwen3.6-27B, supervised by 80 professional annotators under blind labeling and triple-review protocols, that scores every image across all 56 facets with fine-grained, rubric-grounded diagnostics. Empirically, Qwen-Image-Bench reliably distinguishes leading T2I models, achieving the greatest separation on the application-driven dimensions where existing benchmarks provide limited insight. %U https://aclanthology.org/2026.emnlp-main.988/ %P 22760-22787
Markdown (Informal)
[Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation](https://aclanthology.org/2026.emnlp-main.988/) (Li et al., EMNLP 2026)
- Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation (Li et al., EMNLP 2026)
ACL
- Niantong Li, Guangzheng Hu, Weixu Qiao, Ying Ba, Qichen Hong, Shijun Shen, Jinlin Wang, Fan Zhou, Jianye Kang, Xin Shang, Ziyi HE, Wei Wang, Dalin Li, Jiahao Li, Jie Zhang, Kaiyuan Gao, Kun Yan, Lihan Jiang, Ningyuan Tang, Shengming Yin, Tianhe Wu, Xiao Xu, Xiaoyue Chen, Yuxiang Chen, Yan Shu, Yanran Zhang, Yilei Chen, Yixian Xu, Zekai Zhang, Zhendong Wang, Zihao Liu, Zikai Zhou, Hongzhu Shi, Yi Wang, Bing Zhao, hu Wei, Lin Qu, and Chenfei Wu. 2026. Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 22760–22787, Budapest, Hungary. Association for Computational Linguistics.