Word Segmentation for Vietnamese Text Categorization An Internet-based Statistic and Genetic Algorithm Approach

Hung Nguyen Thanh, Khanh Bui Doan


Abstract
This paper suggests a novel Vietnamese segmentation approach for text categorization. Instead of using an annotated training corpus or a lexicon which are still lacking in Vietnamese, we use both statistical information extracted directly from a commercial search engine and a genetic algorithm to find the optimal routes to segmentation. The extracted information includes document frequency and n-gram mutual information. Our experiment results obtained on the segmentation and categorization of online news abstracts are very promising. It matches near 80 % human judgment on segmentation and over 90 % micro-averaging F1 in categorization. The processing time is less than one second per document when statistical information is cached.
Anthology ID:
2006.jeptalnrecital-poster.20
Volume:
Actes de la 13ème conférence sur le Traitement Automatique des Langues Naturelles. Posters
Month:
April
Year:
2006
Address:
Leuven, Belgique
Venue:
JEP/TALN/RECITAL
SIG:
Publisher:
ATALA
Note:
Pages:
561–570
Language:
URL:
https://aclanthology.org/2006.jeptalnrecital-poster.20
DOI:
Bibkey:
Cite (ACL):
Hung Nguyen Thanh and Khanh Bui Doan. 2006. Word Segmentation for Vietnamese Text Categorization An Internet-based Statistic and Genetic Algorithm Approach. In Actes de la 13ème conférence sur le Traitement Automatique des Langues Naturelles. Posters, pages 561–570, Leuven, Belgique. ATALA.
Cite (Informal):
Word Segmentation for Vietnamese Text Categorization An Internet-based Statistic and Genetic Algorithm Approach (Nguyen Thanh & Bui Doan, JEP/TALN/RECITAL 2006)
Copy Citation:
PDF:
https://aclanthology.org/2006.jeptalnrecital-poster.20.pdf