Miyu Oba
2024
Modeling Overregularization in Children with Small Language Models
Akari Haga
|
Saku Sugawara
|
Akiyo Fukatsu
|
Miyu Oba
|
Hiroki Ouchi
|
Taro Watanabe
|
Yohei Oseki
Findings of the Association for Computational Linguistics ACL 2024
The imitation of the children’s language acquisition process has been explored to make language models (LMs) more efficient.In particular, errors caused by children’s regularization (so-called overregularization, e.g., using wroted for the past tense of write) have been widely studied to reveal the mechanisms of language acquisition. Existing research has analyzed regularization in language acquisition only by modeling word inflection directly, which is unnatural in light of human language acquisition. In this paper, we hypothesize that language models that imitate the errors children make during language acquisition have a learning process more similar to humans. To verify this hypothesis, we analyzed the learning curve and error preferences of verb inflections in small-scale LMs using acceptability judgments. We analyze the differences in results by model architecture, data, and tokenization. Our model shows child-like U-shaped learning curves clearly for certain verbs, but the preferences for types of overgeneralization did not fully match the observations in children.
2023
Second Language Acquisition of Neural Language Models
Miyu Oba
|
Tatsuki Kuribayashi
|
Hiroki Ouchi
|
Taro Watanabe
Findings of the Association for Computational Linguistics: ACL 2023
With the success of neural language models (LMs), their language acquisition has gained much attention. This work sheds light on the second language (L2) acquisition of LMs, while previous work has typically explored their first language (L1) acquisition. Specifically, we trained bilingual LMs with a scenario similar to human L2 acquisition and analyzed their cross-lingual transfer from linguistic perspectives. Our exploratory experiments demonstrated that the L1 pretraining accelerated their linguistic generalization in L2, and language transfer configurations (e.g., the L1 choice, and presence of parallel texts) substantially affected their generalizations. These clarify their (non-)human-like L2 acquisition in particular aspects.
BabyLM Challenge: Curriculum learning based on sentence complexity approximating language acquisition
Miyu Oba
|
Akari Haga
|
Akiyo Fukatsu
|
Yohei Oseki
Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning
Search
Co-authors
- Hiroki Ouchi 2
- Taro Watanabe 2
- Akari Haga 2
- Akiyo Fukatsu 2
- Yohei Oseki 2
- show all...