Streaming Language-Specific Twitter Data with Optimal Keywords

Tim Kreutz, Walter Daelemans


Abstract
The Twitter Streaming API has been used to create language-specific corpora with varying degrees of success. Selecting a filter of frequent yet distinct keywords for German resulted in a near-complete collection of German tweets. This method is promising as it keeps within Twitter endpoint limitations and could be applied to other languages besides German. But so far no research has compared methods for selecting optimal keywords for this task. This paper proposes a method for finding optimal key phrases based on a greedy solution to the maximum coverage problem. We generate candidate key phrases for the 50 most frequent languages on Twitter. Candidates are then iteratively selected based on a variety of scoring functions applied to their coverage of target tweets. Selecting candidates based on the scoring function that exponentiates the precision of a key phrase and weighs it by recall achieved the best results overall. Some target languages yield lower results than what could be expected from their prevalence on Twitter. Upon analyzing the errors, we find that these are languages that are very close to more prevalent languages. In these cases, key phrases that limit finding the competitive language are selected, and overall recall on the target language also decreases. We publish the resulting optimized lists for each language as a resource. The code to generate lists for other research objectives is also supplied.
Anthology ID:
2020.wac-1.8
Volume:
Proceedings of the 12th Web as Corpus Workshop
Month:
May
Year:
2020
Address:
Marseille, France
Editors:
Adrien Barbaresi, Felix Bildhauer, Roland Schäfer, Egon Stemle
Venue:
WAC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
57–64
Language:
English
URL:
https://aclanthology.org/2020.wac-1.8
DOI:
Bibkey:
Cite (ACL):
Tim Kreutz and Walter Daelemans. 2020. Streaming Language-Specific Twitter Data with Optimal Keywords. In Proceedings of the 12th Web as Corpus Workshop, pages 57–64, Marseille, France. European Language Resources Association.
Cite (Informal):
Streaming Language-Specific Twitter Data with Optimal Keywords (Kreutz & Daelemans, WAC 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.wac-1.8.pdf