Gregory Marton


2006

pdf bib
Nuggeteer: Automatic Nugget-Based Evaluation using Descriptions and Judgements
Gregory Marton | Alexey Radul
Proceedings of the Human Language Technology Conference of the NAACL, Main Conference

pdf bib
Using Semantic Overlap Scoring in Answering TREC Relationship Questions
Gregory Marton | Boris Katz
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)

A first step in answering complex questions, such as those in the “Relationship”' task of the Text REtrieval Conference's Question Answering track (TREC/QA), is finding passages likely to contain pieces of the answer---passage retrieval. We introduce semantic overlap scoring, a new passage retrieval algorithm that facilitates credit assignment for inexact matches between query and candidate answer. Our official submission ranked best among fully automatic systems, at 23% F-measure, while the best system, with manual input, reached 28%. We use our Nuggeteer tool to robustly evaluate each component of our Relationship system post hoc. Ablation studies show that semantic overlap scoring achieves significant performance improvements over a standard passage retrieval baseline.