Abstract
This paper presents a case study of applying machine translation quality estimation (QE) for the purpose of machine translation (MT) engine selection. The goal is to understand how well the QE predictions correlate with several MT evaluation metrics (automatic and human). Our findings show that our industry-level QE system is not reliable enough for MT selection when the MT systems have similar performance. We suggest that QE can be used with more success for other tasks relevant for translation industry such as risk prevention.- Anthology ID:
- 2020.eamt-1.36
- Volume:
- Proceedings of the 22nd Annual Conference of the European Association for Machine Translation
- Month:
- November
- Year:
- 2020
- Address:
- Lisboa, Portugal
- Editors:
- André Martins, Helena Moniz, Sara Fumega, Bruno Martins, Fernando Batista, Luisa Coheur, Carla Parra, Isabel Trancoso, Marco Turchi, Arianna Bisazza, Joss Moorkens, Ana Guerberof, Mary Nurminen, Lena Marg, Mikel L. Forcada
- Venue:
- EAMT
- SIG:
- Publisher:
- European Association for Machine Translation
- Note:
- Pages:
- 339–346
- Language:
- URL:
- https://aclanthology.org/2020.eamt-1.36
- DOI:
- Bibkey:
- Cite (ACL):
- Anna Zaretskaya, José Conceição, and Frederick Bane. 2020. Estimation vs Metrics: is QE Useful for MT Model Selection?. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 339–346, Lisboa, Portugal. European Association for Machine Translation.
- Cite (Informal):
- Estimation vs Metrics: is QE Useful for MT Model Selection? (Zaretskaya et al., EAMT 2020)
- Copy Citation:
- PDF:
- https://aclanthology.org/2020.eamt-1.36.pdf
Export citation
@inproceedings{zaretskaya-etal-2020-estimation, title = "Estimation vs Metrics: is {QE} Useful for {MT} Model Selection?", author = "Zaretskaya, Anna and Concei{\c{c}}{\~a}o, Jos{\'e} and Bane, Frederick", editor = "Martins, Andr{\'e} and Moniz, Helena and Fumega, Sara and Martins, Bruno and Batista, Fernando and Coheur, Luisa and Parra, Carla and Trancoso, Isabel and Turchi, Marco and Bisazza, Arianna and Moorkens, Joss and Guerberof, Ana and Nurminen, Mary and Marg, Lena and Forcada, Mikel L.", booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation", month = nov, year = "2020", address = "Lisboa, Portugal", publisher = "European Association for Machine Translation", url = "https://aclanthology.org/2020.eamt-1.36", pages = "339--346", abstract = "This paper presents a case study of applying machine translation quality estimation (QE) for the purpose of machine translation (MT) engine selection. The goal is to understand how well the QE predictions correlate with several MT evaluation metrics (automatic and human). Our findings show that our industry-level QE system is not reliable enough for MT selection when the MT systems have similar performance. We suggest that QE can be used with more success for other tasks relevant for translation industry such as risk prevention.", }
<?xml version="1.0" encoding="UTF-8"?> <modsCollection xmlns="http://www.loc.gov/mods/v3"> <mods ID="zaretskaya-etal-2020-estimation"> <titleInfo> <title>Estimation vs Metrics: is QE Useful for MT Model Selection?</title> </titleInfo> <name type="personal"> <namePart type="given">Anna</namePart> <namePart type="family">Zaretskaya</namePart> <role> <roleTerm authority="marcrelator" type="text">author</roleTerm> </role> </name> <name type="personal"> <namePart type="given">José</namePart> <namePart type="family">Conceição</namePart> <role> <roleTerm authority="marcrelator" type="text">author</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Frederick</namePart> <namePart type="family">Bane</namePart> <role> <roleTerm authority="marcrelator" type="text">author</roleTerm> </role> </name> <originInfo> <dateIssued>2020-11</dateIssued> </originInfo> <typeOfResource>text</typeOfResource> <relatedItem type="host"> <titleInfo> <title>Proceedings of the 22nd Annual Conference of the European Association for Machine Translation</title> </titleInfo> <name type="personal"> <namePart type="given">André</namePart> <namePart type="family">Martins</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Helena</namePart> <namePart type="family">Moniz</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Sara</namePart> <namePart type="family">Fumega</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Bruno</namePart> <namePart type="family">Martins</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Fernando</namePart> <namePart type="family">Batista</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Luisa</namePart> <namePart type="family">Coheur</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Carla</namePart> <namePart type="family">Parra</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Isabel</namePart> <namePart type="family">Trancoso</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Marco</namePart> <namePart type="family">Turchi</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Arianna</namePart> <namePart type="family">Bisazza</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Joss</namePart> <namePart type="family">Moorkens</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Ana</namePart> <namePart type="family">Guerberof</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Mary</namePart> <namePart type="family">Nurminen</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Lena</namePart> <namePart type="family">Marg</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Mikel</namePart> <namePart type="given">L</namePart> <namePart type="family">Forcada</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <originInfo> <publisher>European Association for Machine Translation</publisher> <place> <placeTerm type="text">Lisboa, Portugal</placeTerm> </place> </originInfo> <genre authority="marcgt">conference publication</genre> </relatedItem> <abstract>This paper presents a case study of applying machine translation quality estimation (QE) for the purpose of machine translation (MT) engine selection. The goal is to understand how well the QE predictions correlate with several MT evaluation metrics (automatic and human). Our findings show that our industry-level QE system is not reliable enough for MT selection when the MT systems have similar performance. We suggest that QE can be used with more success for other tasks relevant for translation industry such as risk prevention.</abstract> <identifier type="citekey">zaretskaya-etal-2020-estimation</identifier> <location> <url>https://aclanthology.org/2020.eamt-1.36</url> </location> <part> <date>2020-11</date> <extent unit="page"> <start>339</start> <end>346</end> </extent> </part> </mods> </modsCollection>
%0 Conference Proceedings %T Estimation vs Metrics: is QE Useful for MT Model Selection? %A Zaretskaya, Anna %A Conceição, José %A Bane, Frederick %Y Martins, André %Y Moniz, Helena %Y Fumega, Sara %Y Martins, Bruno %Y Batista, Fernando %Y Coheur, Luisa %Y Parra, Carla %Y Trancoso, Isabel %Y Turchi, Marco %Y Bisazza, Arianna %Y Moorkens, Joss %Y Guerberof, Ana %Y Nurminen, Mary %Y Marg, Lena %Y Forcada, Mikel L. %S Proceedings of the 22nd Annual Conference of the European Association for Machine Translation %D 2020 %8 November %I European Association for Machine Translation %C Lisboa, Portugal %F zaretskaya-etal-2020-estimation %X This paper presents a case study of applying machine translation quality estimation (QE) for the purpose of machine translation (MT) engine selection. The goal is to understand how well the QE predictions correlate with several MT evaluation metrics (automatic and human). Our findings show that our industry-level QE system is not reliable enough for MT selection when the MT systems have similar performance. We suggest that QE can be used with more success for other tasks relevant for translation industry such as risk prevention. %U https://aclanthology.org/2020.eamt-1.36 %P 339-346
Markdown (Informal)
[Estimation vs Metrics: is QE Useful for MT Model Selection?](https://aclanthology.org/2020.eamt-1.36) (Zaretskaya et al., EAMT 2020)
- Estimation vs Metrics: is QE Useful for MT Model Selection? (Zaretskaya et al., EAMT 2020)
ACL
- Anna Zaretskaya, José Conceição, and Frederick Bane. 2020. Estimation vs Metrics: is QE Useful for MT Model Selection?. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 339–346, Lisboa, Portugal. European Association for Machine Translation.