Immersive Language Exploration with Object Recognition and Augmented Reality

Benny Platte, Anett Platte, Christian Roschke, Rico Thomanek, Thony Rolletschke, Frank Zimmer, Marc Ritter


Abstract
The use of Augmented Reality (AR) in teaching and learning contexts for language is still young. The ideas are endless, the concrete educational offers available emerge only gradually. Educational opportunities that were unthinkable a few years ago are now feasible. We present a concrete realization: an executable application for mobile devices with which users can explore their environment interactively in different languages. The software recognizes up to 1000 objects in the user’s environment using a deep learning method based on Convolutional Neural Networks and names this objects accordingly. Using Augmented Reality the objects are superimposed with 3D information in different languages. By switching the languages, the user is able to interactively discover his surrounding everyday items in all languages. The application is available as Open Source.
Anthology ID:
2020.lrec-1.44
Volume:
Proceedings of the 12th Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
356–362
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.44
DOI:
Bibkey:
Cite (ACL):
Benny Platte, Anett Platte, Christian Roschke, Rico Thomanek, Thony Rolletschke, Frank Zimmer, and Marc Ritter. 2020. Immersive Language Exploration with Object Recognition and Augmented Reality. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 356–362, Marseille, France. European Language Resources Association.
Cite (Informal):
Immersive Language Exploration with Object Recognition and Augmented Reality (Platte et al., LREC 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.lrec-1.44.pdf