A Comparative Study of Text Preprocessing Approaches for Topic Detection of User Utterances

Roman Sergienko, Muhammad Shan, Wolfgang Minker


Abstract
The paper describes a comparative study of existing and novel text preprocessing and classification techniques for domain detection of user utterances. Two corpora are considered. The first one contains customer calls to a call centre for further call routing; the second one contains answers of call centre employees with different kinds of customer orientation behaviour. Seven different unsupervised and supervised term weighting methods were applied. The collective use of term weighting methods is proposed for classification effectiveness improvement. Four different dimensionality reduction methods were applied: stop-words filtering with stemming, feature selection based on term weights, feature transformation based on term clustering, and a novel feature transformation method based on terms belonging to classes. As classification algorithms we used k-NN and a SVM-based algorithm. The numerical experiments have shown that the simultaneous use of the novel proposed approaches (collectives of term weighting methods and the novel feature transformation method) allows reaching the high classification results with very small number of features.
Anthology ID:
L16-1288
Volume:
Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)
Month:
May
Year:
2016
Address:
Portorož, Slovenia
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
1826–1831
Language:
URL:
https://aclanthology.org/L16-1288
DOI:
Bibkey:
Cite (ACL):
Roman Sergienko, Muhammad Shan, and Wolfgang Minker. 2016. A Comparative Study of Text Preprocessing Approaches for Topic Detection of User Utterances. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1826–1831, Portorož, Slovenia. European Language Resources Association (ELRA).
Cite (Informal):
A Comparative Study of Text Preprocessing Approaches for Topic Detection of User Utterances (Sergienko et al., LREC 2016)
Copy Citation:
PDF:
https://aclanthology.org/L16-1288.pdf