Gregory Marton
2006
Using Semantic Overlap Scoring in Answering TREC Relationship Questions
Gregory Marton
|
Boris Katz
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)
A first step in answering complex questions, such as those in the Relationship' task of the Text REtrieval Conference's Question Answering track (TREC/QA), is finding passages likely to contain pieces of the answer---passage retrieval. We introduce semantic overlap scoring, a new passage retrieval algorithm that facilitates credit assignment for inexact matches between query and candidate answer. Our official submission ranked best among fully automatic systems, at 23% F-measure, while the best system, with manual input, reached 28%. We use our Nuggeteer tool to robustly evaluate each component of our Relationship system post hoc. Ablation studies show that semantic overlap scoring achieves significant performance improvements over a standard passage retrieval baseline.
Nuggeteer: Automatic Nugget-Based Evaluation using Descriptions and Judgements
Gregory Marton
|
Alexey Radul
Proceedings of the Human Language Technology Conference of the NAACL, Main Conference
Search