NIST 2009 Open Machine Translation Evaluation (MT09)
Official Release of Results

Date of release: Tue Oct 27 15:48:58 2009
Version: mt09_public_v1

Introduction

This release page is limited to the Current Test for the Arabic-to-English track.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Results for submissions from GALE participants, who had previous access to the test data, will be reported separately from results submitted by those that are not part of the GALE program.

Participants

The following table lists the submissions received from the sites participating in the Arabic-to-English Current Test.

Site ID

Organization

Location

Current Test Set, Arabic-to-English

Single System Track

System Combination Track

amsterdam

University of Amsterdam

Netherlands

Yes(1)

-

apptek

AppTek

USA

Yes

-

bbn

BBN Technologies

USA

Yes

-

cmu-smt

Carnegie Mellon LTI interACT

USA

Yes

-

cmu-statxfer

Carnegie Mellon StatXfer

USA

Yes

-

columbia

Columbia University

USA

Yes

-

cued

Cambridge University Engineering Department

UK

Yes

-

edinburgh

University of Edinburgh

UK

Yes

-

fbk

Fondazione Bruno Kessler

Italy

Yes

-

ibm

IBM

USA

Yes

Yes

isi-lw

University of Southern California / Language Weaver Inc.

USA

Yes

Yes

jhu

Johns Hopkins University

USA

Late and/or debugged submission

-

kcsl

KCSL Inc.

Canada

Yes

-

limsi

Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur - CNRS

France

Yes

-

lium-systran

Université du Maine (Le Mans) / SYSTRAN

.

Yes(1)

-

rwth

RWTH-Aachen University, Chair of Computer Sciences

Germany

Yes

-

sakhr

Sakhr Software

Egypt

Yes

-

sri

SRI International

USA

Yes

Yes

stanford

Stanford University

USA

Yes

-

telaviv

Tel Aviv University

Israel

Yes

-

tubitak-uekae

TUBITAK-UEKAE

Turkey

Yes

-

umd

University of Maryland

USA

Yes

-

upc-lsi

UPC-LSI (Universitat Politècnica de Catalunya, Llenguatges i Sistemes Informàtics)

Spain

Yes

-

(1)A late and/or debugged system was also submitted, not reported here.

Current Test Results - Single System Track

Arabic-to-English, submissions from participants involved in GALE (Table 1)

Site ID

System

BLEU-4 (mteval-v13a)

Overall

Newswire

Web

Constrained Data Track

cued

CUED_a2e_cn_primary

0.4834

0.5641

0.3960

 

 

 

stanford

stanford_a2e_cn_primary

0.4781

0.5673

0.3843

 

 

 

isi-lw

isi-lw_a2e_cn_primary

0.4763

0.5590

0.3810

 

 

 

ibm

IBM_a2e_constrained_primary

0.4708

0.5547

0.3833

 

 

 

bbn

BBN_a2e_cn_primary

0.4680

0.5566

0.3783

 

 

 

rwth

RWTH_a2e_cn_primary

0.4534

0.5402

0.3538

 

 

 

sri

SRI_a2e_cn_primary

0.4527

0.5366

0.3634

 

 

 

edinburgh

Edinburgh_a2e_cn_primary

0.4479

0.5240

0.3605

 

 

 

umd

UMD_a2e_cn_primary

0.4409

0.5340

0.3415

 

 

 

cmu-smt

CMU-SMT_a2e_cn_primary

0.4304

0.5055

0.3473

 

 

 

columbia

columbia_a2e_cn_primary

0.4157

0.4932

0.3331

 

 

 

cmu-statxfer

CMU-Stat-Xfer_a2e_cn_primary

0.3774

0.4448

0.2986

 

 

 

sakhr

SAKHR_a2e_cn_primary

0.3681

0.4185

0.3147

 

 

 

UnConstrained Data Track

ibm

IBM_arabic_un_primary

0.4981

0.5713

0.4214

 

 

 

apptek

AppTek_a2e_un_primary

0.4790

0.5165

0.4352

 

 

 

Arabic-to-English, submissions from participants not involved in GALE (Table 2)

Site ID

System

BLEU-4 (mteval-v13a)

Overall

Newswire

Web

Constrained Data Track

lium-systran

LIUM-SYSTRAN_a2e_cn_primary(1)

0.4773

0.5629

0.3800

 

 

 

fbk

FBK_a2e_cn_primary

0.4567

0.5418

0.3615

 

 

 

limsi

LIMSI_Moses_a2e_cn_primary

0.4384

0.5242

0.3471

 

 

 

tubitak-uekae

TUBITAK_a2e_cn_primary

0.4112

0.4826

0.3310

 

 

 

upc-lsi

UPC.LSI_a2e_cn_primary

0.3588

0.4344

0.2778

 

 

 

amsterdam

UvA_a2e_cn_primary(1)

0.3221

0.3820

0.2565

 

 

 

kcsl

KCSL_a2e_cn_primary

0.1422

0.1670

0.1161

 

 

 

telaviv

TLVEBMT_a2e_cn_primary

0.0703

0.0872

0.0527

 

 

 

(1)A late and/or debugged system was also submitted, not reported here.

Current Test Results - System Combination Track

Arabic-to-English, submissions from participants involved in GALE (Table 3)

Site ID

System

BLEU-4 (mteval-v13a)

Overall

Newswire

Web

Constrained Data Track

isi-lw

isi-lw_a2e_cn_combo1

0.4802

0.5600

0.3914

 

 

 

ibm

IBM_a2e_cn_combo0

0.4775

0.5636

0.3871

 

 

 

sri

SRI_a2e_cn_combo1

0.4631

0.5472

0.3731

 

 

 

UnConstrained Data Track

ibm

IBM_a2e_un_combo0

0.5096

0.5913

0.4241

 

 

 

Current Test Results - Single System Track and System Combination Track - All metrics

Arabic-to-English, submissions from participants involved in GALE (Table 4)

Site ID

System

BLEU-4 (mteval-v13a)

IBM BLEU (bleu-1.04)

NIST (mteval-v13a)

TER (tercom-0.7.25)

METEOR (meteor-0.7)

Overall

Newswire

Web

Overall

Newswire

Web

Overall

Newswire

Web

Overall

Newswire

Web

Overall

Newswire

Web

Constrained Data Track

cued

CUED_a2e_cn_primary

0.4834

0.5641

0.3960

0.4833

0.5640

0.3959

11.01

11.31

9.603

0.4489

0.3861

0.5093

0.6570

0.7152

0.5999

isi-lw

isi-lw_a2e_cn_combo1

0.4802

0.5600

0.3914

0.4801

0.5598

0.3913

10.85

11.24

9.396

0.4643

0.3887

0.5371

0.6642

0.7319

0.5969

stanford

stanford_a2e_cn_primary

0.4781

0.5673

0.3843

0.4777

0.5668

0.3840

10.97

11.44

9.392

0.4399

0.3709

0.5065

0.6514

0.7153

0.5882

ibm

IBM_a2e_cn_combo0

0.4775

0.5636

0.3871

0.4773

0.5634

0.3870

11.03

11.54

9.466

0.4394

0.3713

0.5051

0.6526

0.7142

0.5924

isi-lw

isi-lw_a2e_cn_primary

0.4763

0.5590

0.3810

0.4760

0.5588

0.3808

10.85

11.30

9.292

0.4590

0.3826

0.5328

0.6544

0.7233

0.5864

ibm

IBM_a2e_constrained_primary

0.4708

0.5547

0.3833

0.4707

0.5545

0.3831

10.97

11.46

9.406

0.4424

0.3773

0.5051

0.6478

0.7091

0.5876

bbn

BBN_a2e_cn_primary

0.4680

0.5566

0.3783

0.4678

0.5564

0.3781

10.85

11.46

9.304

0.4603

0.3800

0.5378

0.6561

0.7125

0.6015

sri

SRI_a2e_cn_combo1

0.4631

0.5472

0.3731

0.4629

0.5470

0.3730

10.86

11.25

9.306

0.4592

0.3974

0.5189

0.6461

0.7063

0.5866

rwth

RWTH_a2e_cn_primary

0.4534

0.5402

0.3538

0.4533

0.5400

0.3537

10.65

11.12

9.028

0.4666

0.3980

0.5328

0.6532

0.7154

0.5918

sri

SRI_a2e_cn_primary

0.4527

0.5366

0.3634

0.4526

0.5365

0.3633

10.71

11.11

9.251

0.4689

0.4003

0.5351

0.6459

0.7071

0.5859

edinburgh

Edinburgh_a2e_cn_primary

0.4479

0.5240

0.3605

0.4478

0.5238

0.3604

10.58

10.92

9.174

0.4857

0.4209

0.5482

0.6454

0.7069

0.5847

umd

UMD_a2e_cn_primary

0.4409

0.5340

0.3415

0.4408

0.5338

0.3413

10.53

11.25

8.590

0.4591

0.3909

0.5248

0.6287

0.6982

0.5595

cmu-smt

CMU-SMT_a2e_cn_primary

0.4304

0.5055

0.3473

0.4302

0.5053

0.3469

10.33

10.72

8.896

0.4823

0.4182

0.5442

0.6365

0.6988

0.5748

columbia

columbia_a2e_cn_primary

0.4157

0.4932

0.3331

0.4156

0.4931

0.3330

10.27

10.73

8.785

0.4747

0.4160

0.5313

0.6269

0.6847

0.5698

cmu-statxfer

CMU-Stat-Xfer_a2e_cn_primary

0.3774

0.4448

0.2986

0.3772

0.4447

0.2984

9.731

10.07

8.399

0.5158

0.4628

0.5669

0.6082

0.6684

0.5484

sakhr

SAKHR_a2e_cn_primary

0.3681

0.4185

0.3147

0.3680

0.4184

0.3147

9.867

9.982

8.887

0.5075

0.4626

0.5508

0.6320

0.6737

0.5913

UnConstrained Data Track

ibm

IBM_a2e_un_combo0

0.5096

0.5913

0.4241

0.5095

0.5912

0.4240

11.54

11.92

10.02

0.4168

0.3507

0.4804

0.6768

0.7366

0.6181

ibm

IBM_arabic_un_primary

0.4981

0.5713

0.4214

0.4979

0.5711

0.4214

11.41

11.70

9.973

0.4253

0.3653

0.4833

0.6700

0.7293

0.6117

apptek

AppTek_a2e_un_primary

0.4790

0.5165

0.4352

0.4787

0.5162

0.4348

11.21

10.98

10.32

0.4342

0.3969

0.4702

0.6818

0.7207

0.6433

Arabic-to-English, submissions from participants not involved in GALE (Table 5)

Site ID

System

BLEU-4 (mteval-v13a)

IBM BLEU (bleu-1.04)

NIST (mteval-v13a)

TER (tercom-0.7.25)

METEOR (meteor-0.7)

Overall

Newswire

Web

Overall

Newswire

Web

Overall

Newswire

Web

Overall

Newswire

Web

Overall

Newswire

Web

Constrained Data Track

lium-systran

LIUM-SYSTRAN_a2e_cn_primary(1)

0.4773

0.5629

0.3800

0.4772

0.5627

0.3799

10.96

11.38

9.412

0.4565

0.3851

0.5252

0.6526

0.7185

0.5879

fbk

FBK_a2e_cn_primary

0.4567

0.5418

0.3615

0.4565

0.5417

0.3613

10.75

11.22

9.252

0.4721

0.3952

0.5462

0.6533

0.7170

0.5910

limsi

LIMSI_Moses_a2e_cn_primary

0.4384

0.5242

0.3471

0.4383

0.5240

0.3469

10.40

10.98

8.738

0.4724

0.4051

0.5373

0.6212

0.6851

0.5582

tubitak-uekae

TUBITAK_a2e_cn_primary

0.4112

0.4826

0.3310

0.4121

0.4827

0.3312

10.01

10.43

8.659

0.5129

0.4445

0.5789

0.6259

0.6890

0.5627

upc-lsi

UPC.LSI_a2e_cn_primary

0.3588

0.4344

0.2778

0.3588

0.4345

0.2777

9.404

10.05

7.655

0.5188

0.4630

0.5725

0.5866

0.6515

0.5216

amsterdam

UvA_a2e_cn_primary(1)

0.3221

0.3820

0.2565

0.3218

0.3816

0.2563

8.621

9.063

7.458

0.5995

0.5437

0.6533

0.5863

0.6457

0.5278

kcsl

KCSL_a2e_cn_primary

0.1422

0.1670

0.1161

0.1420

0.1669

0.1158

6.647

6.959

5.820

0.6590

0.6353

0.6818

0.4937

0.5398

0.4482

telaviv

TLVEBMT_a2e_cn_primary

0.0703

0.0872

0.0527

0.0703

0.0872

0.0526

3.879

4.370

3.165

0.7483

0.7274

0.7685

0.3853

0.4262

0.3450

(1)A late and/or debugged system was also submitted, not reported here.