@inproceedings{huidrom-etal-2026-aura,
title = "Aura đź’«: A Self-Hostable Platform for Human Evaluation of {NLG} for Under-resourced Languages and Beyond",
author = "Huidrom, Rudali and
Nelson, Khangembam Alex D and
Khumukcham, Nicholas and
Pangsatabam, Hoomexsun and
L, Jesu Nazarene Roy and
Trivedi, Kanal and
Kumar, Vikas and
Das, Pinaki and
Singh, Kshetrimayum Boynao and
Bisht, Tarun and
Konjengbam, Justice and
Anand, Konjengbam and
Goyal, Vikram",
editor = "Zhang, Ningyu and
Wijaya, Derry",
booktitle = "Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the {A}sia-Pacific Chapter of the {A}ssociation for {C}omputational {L}inguistics: System Demonstrations",
month = nov,
year = "2026",
address = "Hengqin, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.aacl-demo.2/",
pages = "9--16",
ISBN = "979-8-89176-441-5",
abstract = "Human evaluation is the gold standard for Natural Language Generation (NLG), but the tooling is fragmented: most platforms serve a single task or a single protocol, and none treat the growing practice of assisting annotators with an LLM as a measurable object of study. These gaps fall hardest on indigenous and under-resourced languages, where paid crowds are scarce, references are unavailable, and data governance matters. We present Aura, an open, self-hostable platform that unifies five evaluation protocols (Likert/DA, pairwise, ranking, error-span, post-editing) across eight task families under one segment schema and one result model, records writing-system and speaker provenance as first-class fields, and runs entirely offline. Its distinguishing feature is a grounded, advisory-only, logged LLM assistant paired with a built-in randomised experiment that estimates the effect of AI assistance on inter-annotator agreement and time-on-task. Aura{'}s agreement statistics match reference implementations to machine precision. We conducted a human evaluation experiment on three English{\textrightarrow}Indian-language ranking campaigns (N=9 annotators) which returned a mean usability rating of 4.06/5. Aura is released as open-source software."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="huidrom-etal-2026-aura">
<titleInfo>
<title>Aura đź’«: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond</title>
</titleInfo>
<name type="personal">
<namePart type="given">Rudali</namePart>
<namePart type="family">Huidrom</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Khangembam</namePart>
<namePart type="given">Alex</namePart>
<namePart type="given">D</namePart>
<namePart type="family">Nelson</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Nicholas</namePart>
<namePart type="family">Khumukcham</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Hoomexsun</namePart>
<namePart type="family">Pangsatabam</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jesu</namePart>
<namePart type="given">Nazarene</namePart>
<namePart type="given">Roy</namePart>
<namePart type="family">L</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Kanal</namePart>
<namePart type="family">Trivedi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Vikas</namePart>
<namePart type="family">Kumar</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Pinaki</namePart>
<namePart type="family">Das</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Kshetrimayum</namePart>
<namePart type="given">Boynao</namePart>
<namePart type="family">Singh</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Tarun</namePart>
<namePart type="family">Bisht</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Justice</namePart>
<namePart type="family">Konjengbam</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Konjengbam</namePart>
<namePart type="family">Anand</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Vikram</namePart>
<namePart type="family">Goyal</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-11</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations</title>
</titleInfo>
<name type="personal">
<namePart type="given">Ningyu</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Derry</namePart>
<namePart type="family">Wijaya</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Hengqin, China</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-441-5</identifier>
</relatedItem>
<abstract>Human evaluation is the gold standard for Natural Language Generation (NLG), but the tooling is fragmented: most platforms serve a single task or a single protocol, and none treat the growing practice of assisting annotators with an LLM as a measurable object of study. These gaps fall hardest on indigenous and under-resourced languages, where paid crowds are scarce, references are unavailable, and data governance matters. We present Aura, an open, self-hostable platform that unifies five evaluation protocols (Likert/DA, pairwise, ranking, error-span, post-editing) across eight task families under one segment schema and one result model, records writing-system and speaker provenance as first-class fields, and runs entirely offline. Its distinguishing feature is a grounded, advisory-only, logged LLM assistant paired with a built-in randomised experiment that estimates the effect of AI assistance on inter-annotator agreement and time-on-task. Aura’s agreement statistics match reference implementations to machine precision. We conducted a human evaluation experiment on three English→Indian-language ranking campaigns (N=9 annotators) which returned a mean usability rating of 4.06/5. Aura is released as open-source software.</abstract>
<identifier type="citekey">huidrom-etal-2026-aura</identifier>
<location>
<url>https://aclanthology.org/2026.aacl-demo.2/</url>
</location>
<part>
<date>2026-11</date>
<extent unit="page">
<start>9</start>
<end>16</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T Aura đź’«: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond
%A Huidrom, Rudali
%A Nelson, Khangembam Alex D.
%A Khumukcham, Nicholas
%A Pangsatabam, Hoomexsun
%A L, Jesu Nazarene Roy
%A Trivedi, Kanal
%A Kumar, Vikas
%A Das, Pinaki
%A Singh, Kshetrimayum Boynao
%A Bisht, Tarun
%A Konjengbam, Justice
%A Anand, Konjengbam
%A Goyal, Vikram
%Y Zhang, Ningyu
%Y Wijaya, Derry
%S Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations
%D 2026
%8 November
%I Association for Computational Linguistics
%C Hengqin, China
%@ 979-8-89176-441-5
%F huidrom-etal-2026-aura
%X Human evaluation is the gold standard for Natural Language Generation (NLG), but the tooling is fragmented: most platforms serve a single task or a single protocol, and none treat the growing practice of assisting annotators with an LLM as a measurable object of study. These gaps fall hardest on indigenous and under-resourced languages, where paid crowds are scarce, references are unavailable, and data governance matters. We present Aura, an open, self-hostable platform that unifies five evaluation protocols (Likert/DA, pairwise, ranking, error-span, post-editing) across eight task families under one segment schema and one result model, records writing-system and speaker provenance as first-class fields, and runs entirely offline. Its distinguishing feature is a grounded, advisory-only, logged LLM assistant paired with a built-in randomised experiment that estimates the effect of AI assistance on inter-annotator agreement and time-on-task. Aura’s agreement statistics match reference implementations to machine precision. We conducted a human evaluation experiment on three English→Indian-language ranking campaigns (N=9 annotators) which returned a mean usability rating of 4.06/5. Aura is released as open-source software.
%U https://aclanthology.org/2026.aacl-demo.2/
%P 9-16
Markdown (Informal)
[Aura đź’«: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond](https://aclanthology.org/2026.aacl-demo.2/) (Huidrom et al., AACL-IJCNLP 2026)
ACL
- Rudali Huidrom, Khangembam Alex D Nelson, Nicholas Khumukcham, Hoomexsun Pangsatabam, Jesu Nazarene Roy L, Kanal Trivedi, Vikas Kumar, Pinaki Das, Kshetrimayum Boynao Singh, Tarun Bisht, Justice Konjengbam, Konjengbam Anand, and Vikram Goyal. 2026. Aura 💫: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond. In Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations, pages 9–16, Hengqin, China. Association for Computational Linguistics.