Language Models as Context-sensitive Word Search Engines

Matti Wiegmann, Michael Völske, Benno Stein, Martin Potthast


Abstract
Context-sensitive word search engines are writing assistants that support word choice, phrasing, and idiomatic language use by indexing large-scale n-gram collections and implementing a wildcard search. However, search results become unreliable with increasing context size (e.g., n>=5), when observations become sparse. This paper proposes two strategies for word search with larger n, based on masked and conditional language modeling. We build such search engines using BERT and BART and compare their capabilities in answering English context queries with those of the n-gram-based word search engine Netspeak. Our proposed strategies score within 5 percentage points MRR of n-gram collections while answering up to 5 times as many queries.
Anthology ID:
2022.in2writing-1.5
Volume:
Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
In2Writing
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
39–45
Language:
URL:
https://aclanthology.org/2022.in2writing-1.5
DOI:
10.18653/v1/2022.in2writing-1.5
Bibkey:
Cite (ACL):
Matti Wiegmann, Michael Völske, Benno Stein, and Martin Potthast. 2022. Language Models as Context-sensitive Word Search Engines. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pages 39–45, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Language Models as Context-sensitive Word Search Engines (Wiegmann et al., In2Writing 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.in2writing-1.5.pdf
Video:
 https://aclanthology.org/2022.in2writing-1.5.mp4
Code
 webis-de/in2writing22-language-models-as-context-sensitive-word-search-engines
Data
CLOTHWikiText-103WikiText-2