NIST 2009 Open Machine Translation Evaluation (MT09)
Official Release of Results

Date of release: Tue Oct 27 15:48:58 2009
Version: mt09_public_v1

Introduction

This page of results is limited to those of the Progress Test data for the MT09 tests of Arabic-to-English and Chinese-to-English text-to-text translation.

The Progress test was designed as a means to demonstrate true system improvement for a particular site over time, irrespective of inherent data differences that come with each new test set. In order to keep the focus on progress over time, only results from sites that participated in both OpenMT08 and OpenMT09 are reported.

There may be issues related to processing older test sets that make across-site comparisons less meaningful.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Note that all BLEU-4 scores reported on this page were computed using version 13a of the mteval script. Last year, scores were computed using mteval-v11b, which uses a different brevity penalty. This year's script (mteval-v13a) uses the 'closest reference translation length' (same as in IBM BLEU) instead of the 'shortest reference translation length', to compute the brevity penalty.

Participants

The following table lists the submissions received from the sites participating in the Progress Test. Cells in bold indicate submissions for which scores are reported:

Site ID

Organization

Location

Progress Test Set

Arabic-to-English

Chinese-to-English

amsterdam

University of Amsterdam

Netherlands

MT09 only

MT09 only

apptek

AppTek

USA

MT08 and MT09

Different training condition

bbn

BBN Technologies

USA

MT08 and MT09

MT08 and MT09

buaa

Beihang University, Institute of Intelligent Information Processing, School of Computer Science and Engineering

China

-

MT09 only

cas-ia

Chinese Academy of Sciences, Institute of Automation

China

-

MT09 only

cas-ict

Chinese Academy of Sciences, Institute of Computing Technology

China

-

MT09 only

ccid

China Center for Information Industry Development

China

-

MT09 only

cmu-smt

Carnegie Mellon LTI interACT

USA

Different training condition

Different training condition

cmu-statxfer

Carnegie Mellon StatXfer

USA

MT09 only

-

dcu

Dublin City University

Ireland

-

MT09 only

dfki

DFKI GmbH

Germany

-

MT09 only

edinburgh

University of Edinburgh

UK

MT09 only

-

fbk

Fondazione Bruno Kessler

Italy

MT09 only

-

frdc

Fujitsu Research & Development Center Co., Ltd.

China

-

MT09 only

hit-ltrc

Harbin Institute of Technology, Language Technology Research Center

China

-

MT09 only

isi-lw

University of Southern California / Language Weaver Inc.

USA

MT08 and MT09

MT08 and MT09

lium

Université du Maine (Le Mans)

France

-

MT09 only

lium-systran

Université du Maine (Le Mans) / SYSTRAN

.

Late and/or debugged submission

-

nju-nlp

Nanjing University NLP

China

-

MT09 only

nrc

National Research Council Canada

Canada

-

MT08 and MT09

nthu

National Tsing Hua University, Department of Computer Science

Taiwan

-

MT09 only

rwth

RWTH-Aachen University, Chair of Computer Sciences

Germany

MT09 only

MT09 only

sakhr

Sakhr Software

Egypt

MT09 only

-

sri

SRI International

USA

MT08 and MT09

MT08 and MT09

systran-lium

SYSTRAN / Université du Maine (Le Mans)

.

-

MT09 only

systran-nrc

SYSTRAN / National Research Council Canada

.

-

MT09 only

tubitak-uekae

TUBITAK-UEKAE

Turkey

MT09 only

-

umd

University of Maryland

USA

-

MT08 and MT09

upc-lsi

UPC-LSI (Universitat Politècnica de Catalunya, Llenguatges i Sistemes Informàtics)

Spain

MT08 and MT09

-

Progress Test Results - Single System Track

Arabic-to-English (Table 1)

Site ID

System

BLEU-4 (mteval-v13a)

Overall

Newswire

Web

MT08

MT09

MT08

MT09

MT08

MT09

Constrained Data Track

bbn

BBN-progress_a2e_cn_primary

0.4186(1)

0.4379

0.4655(1)

0.4926

0.3566(1)

0.3678

 

 

 

isi-lw

isi-lw_a2e_cn_primary

0.4030(1)

0.4296

0.4498(1)

0.4748

0.3408(1)

0.3621

 

 

 

sri

SRI_a2e_cn_primary

0.4011(1)

0.4087

0.4558(1)

0.4499

0.3277(1)

0.3551

 

 

 

upc-lsi

UPC.LSI_a2e_cn_primary

0.2956

0.3303

0.3448

0.3769

0.2300

0.2676

 

 

 

UnConstrained Data Track

apptek

AppTek_a2e_un_primary

0.4195

0.3955

0.4245

0.3991

0.4131

0.3909

 

 

 

(1)The MT08 system was a combined system

Chinese-to-English (Table 2)

Site ID

System

BLEU-4 (mteval-v13a)

Overall

Newswire

Web

MT08

MT09

MT08

MT09

MT08

MT09

Constrained Data Track

isi-lw

isi-lw_c2e_cn_primary

0.2990(1)

0.3225

0.3516(1)

0.3628

0.2237(1)

0.2642

 

 

 

bbn

BBN-progress_c2e_cn_primary

0.3055(1)

0.3153

0.3447(1)

0.3481

0.2509(1)

0.2697

 

 

 

nrc

NRC_c2e_cn_primary

0.2480

0.2811

0.2679

0.3130

0.2204

0.2357

 

 

 

sri

SRI_c2e_cn_primary

0.2617(1)

0.2790

0.3028(1)

0.3132

0.2032(1)

0.2303

 

 

 

umd

UMD_c2e_cn_primary

0.2456

0.2500

0.2823

0.2781

0.1936

0.2108

 

 

 

(1)The MT08 system was a combined system