Mark Hopkins


pdf bib
Williams College’s Submission for the Coco4MT 2023 Shared Task
Alex Root | Mark Hopkins
Proceedings of the Second Workshop on Corpus Generation and Corpus Augmentation for Machine Translation

Professional translation is expensive. As a consequence, when developing a translation system in the absence of a pre-existing parallel corpus, it is important to strategically choose sentences to have professionally translated for the training corpus. In our contribution to the Coco4MT 2023 Shared Task, we explore how sentence embeddings can be leveraged to choose an impactful set of sentences to translate. Based on six language pairs of the JHU Bible corpus, we demonstrate that a technique based on SimCSE embeddings outperforms a competitive suite of baselines.

pdf bib
On the Evaluation of Neural Selective Prediction Methods for Natural Language Processing
Zhengyao Gu | Mark Hopkins
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

We provide a survey and empirical comparison of the state-of-the-art in neural selective classification for NLP tasks. We also provide a methodological blueprint, including a novel metric called refinement that provides a calibrated evaluation of confidence functions for selective prediction. Finally, we supply documented, open-source code to support the future development of selective prediction techniques.


pdf bib
Towards More Natural Artificial Languages
Mark Hopkins
Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)

A number of papers have recently argued in favor of using artificially generated languages to investigate the inductive biases of linguistic models, or to develop models for low-resource languages with underrepresented typologies. But the promise of artificial languages comes with a caveat: if these artificial languages are not sufficiently reflective of natural language, then using them as a proxy may lead to inaccurate conclusions. In this paper, we take a step towards increasing the realism of artificial language by introducing a variant of indexed grammars that draw their weights from hierarchical Pitman-Yor processes. We show that this framework generates languages that emulate the statistics of natural language corpora better than the current approach of directly formulating weighted context-free grammars.


pdf bib
Reed at SemEval-2020 Task 9: Fine-Tuning and Bag-of-Words Approaches to Code-Mixed Sentiment Analysis
Vinay Gopalan | Mark Hopkins
Proceedings of the Fourteenth Workshop on Semantic Evaluation

We explore the task of sentiment analysis on Hinglish (code-mixed Hindi-English) tweets as participants of Task 9 of the SemEval-2020 competition, known as the SentiMix task. We had two main approaches: 1) applying transfer learning by fine-tuning pre-trained BERT models and 2) training feedforward neural networks on bag-of-words representations. During the evaluation phase of the competition, we obtained an F-score of 71.3% with our best model, which placed 4th out of 62 entries in the official system rankings.


pdf bib
SemEval-2019 Task 10: Math Question Answering
Mark Hopkins | Ronan Le Bras | Cristian Petrescu-Prahova | Gabriel Stanovsky | Hannaneh Hajishirzi | Rik Koncel-Kedziorski
Proceedings of the 13th International Workshop on Semantic Evaluation

We report on the SemEval 2019 task on math question answering. We provided a question set derived from Math SAT practice exams, including 2778 training questions and 1082 test questions. For a significant subset of these questions, we also provided SMT-LIB logical form annotations and an interpreter that could solve these logical forms. Systems were evaluated based on the percentage of correctly answered questions. The top system correctly answered 45% of the test questions, a considerable improvement over the 17% random guessing baseline.


pdf bib
Spot the Odd Man Out: Exploring the Associative Power of Lexical Resources
Gabriel Stanovsky | Mark Hopkins
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

We propose Odd-Man-Out, a novel task which aims to test different properties of word representations. An Odd-Man-Out puzzle is composed of 5 (or more) words, and requires the system to choose the one which does not belong with the others. We show that this simple setup is capable of teasing out various properties of different popular lexical resources (like WordNet and pre-trained word embeddings), while being intuitive enough to annotate on a large scale. In addition, we propose a novel technique for training multi-prototype word representations, based on unsupervised clustering of ELMo embeddings, and show that it surpasses all other representations on all Odd-Man-Out collections.

pdf bib
Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples
Vidur Joshi | Matthew Peters | Mark Hopkins
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

We revisit domain adaptation for parsers in the neural era. First we show that recent advances in word representations greatly diminish the need for domain adaptation when the target domain is syntactically similar to the source domain. As evidence, we train a parser on the Wall Street Journal alone that achieves over 90% F1 on the Brown corpus. For more syntactically distant domains, we provide a simple way to adapt a parser using only dozens of partial annotations. For instance, we increase the percentage of error-free geometry-domain parses in a held-out set from 45% to 73% using approximately five dozen training examples. In the process, we demonstrate a new state-of-the-art single model result on the Wall Street Journal test set of 94.3%. This is an absolute increase of 1.7% over the previous state-of-the-art of 92.6%.


pdf bib
Beyond Sentential Semantic Parsing: Tackling the Math SAT with a Cascade of Tree Transducers
Mark Hopkins | Cristian Petrescu-Prahova | Roie Levin | Ronan Le Bras | Alvaro Herrasti | Vidur Joshi
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

We present an approach for answering questions that span multiple sentences and exhibit sophisticated cross-sentence anaphoric phenomena, evaluating on a rich source of such questions – the math portion of the Scholastic Aptitude Test (SAT). By using a tree transducer cascade as its basic architecture, our system propagates uncertainty from multiple sources (e.g. coreference resolution or verb interpretation) until it can be confidently resolved. Experiments show the first-ever results 43% recall and 91% precision) on SAT algebra word problems. We also apply our system to the public Dolphin algebra question set, and improve the state-of-the-art F1-score from 73.9% to 77.0%.

pdf bib
Interactive Visualization for Linguistic Structure
Aaron Sarnat | Vidur Joshi | Cristian Petrescu-Prahova | Alvaro Herrasti | Brandon Stilson | Mark Hopkins
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

We provide a visualization library and web interface for interactively exploring a parse tree or a forest of parses. The library is not tied to any particular linguistic representation, but provides a general-purpose API for the interactive exploration of hierarchical linguistic structure. To facilitate rapid understanding of a complex structure, the API offers several important features, including expand/collapse functionality, positional and color cues, explicit visual support for sequential structure, and dynamic highlighting to convey node-to-text correspondence.


pdf bib
Source-side Preordering for Translation using Logistic Regression and Depth-first Branch-and-Bound Search
Laura Jehl | Adrià de Gispert | Mark Hopkins | Bill Byrne
Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics


pdf bib
Models of Translation Competitions
Mark Hopkins | Jonathan May
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)


pdf bib
Extraction Programs: A Unified Approach to Translation Rule Extraction
Mark Hopkins | Greg Langmead | Tai Vo
Proceedings of the Sixth Workshop on Statistical Machine Translation

pdf bib
Tuning as Ranking
Mark Hopkins | Jonathan May
Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing


pdf bib
SCFG Decoding Without Binarization
Mark Hopkins | Greg Langmead
Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing


pdf bib
Cube Pruning as Heuristic Search
Mark Hopkins | Greg Langmead
Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing


pdf bib
Machine Translation as Tree Labeling
Mark Hopkins | Jonas Kuhn
Proceedings of SSST, NAACL-HLT 2007 / AMTA Workshop on Syntax and Structure in Statistical Translation

pdf bib
Deep Grammars in a Tree Labeling Approach to Syntax-based Statistical Machine Translation
Mark Hopkins | Jonas Kuhn
ACL 2007 Workshop on Deep Linguistic Processing


pdf bib
Exploring the Potential of Intractable Parsers
Mark Hopkins | Jonas Kuhn
Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions

pdf bib
A Framework for Incorporating Alignment Information in Parsing
Mark Hopkins | Jonas Kuhn
Proceedings of the Cross-Language Knowledge Induction Workshop


pdf bib
What’s in a translation rule?
Michel Galley | Mark Hopkins | Kevin Knight | Daniel Marcu
Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004