EMISSOR: A platform for capturing multimodal interactions as Episodic Memories and Interpretations with Situated Scenario-based Ontological References

Selene Baez Santamaria, Thomas Baier, Taewoon Kim, Lea Krause, Jaap Kruijt, Piek Vossen


Abstract
We present EMISSOR: a platform to capture multimodal interactions as recordings of episodic experiences with explicit referential interpretations that also yield an episodic Knowledge Graph (eKG). The platform stores streams of multiple modalities as parallel signals. Each signal is segmented and annotated independently with interpretation. Annotations are eventually mapped to explicit identities and relations in the eKG. As we ground signal segments from different modalities to the same instance representations, we also ground different modalities across each other. Unique to our eKG is that it accepts different interpretations across modalities, sources and experiences and supports reasoning over conflicting information and uncertainties that may result from multimodal experiences. EMISSOR can record and annotate experiments in virtual and real-world, combine data, evaluate system behavior and their performance for preset goals but also model the accumulation of knowledge and interpretations in the Knowledge Graph as a result of these episodic experiences.
Anthology ID:
2021.mmsr-1.6
Volume:
Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR)
Month:
June
Year:
2021
Address:
Groningen, Netherlands (Online)
Editors:
Lucia Donatelli, Nikhil Krishnaswamy, Kenneth Lai, James Pustejovsky
Venue:
MMSR
SIG:
SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
56–77
Language:
URL:
https://aclanthology.org/2021.mmsr-1.6
DOI:
Bibkey:
Cite (ACL):
Selene Baez Santamaria, Thomas Baier, Taewoon Kim, Lea Krause, Jaap Kruijt, and Piek Vossen. 2021. EMISSOR: A platform for capturing multimodal interactions as Episodic Memories and Interpretations with Situated Scenario-based Ontological References. In Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR), pages 56–77, Groningen, Netherlands (Online). Association for Computational Linguistics.
Cite (Informal):
EMISSOR: A platform for capturing multimodal interactions as Episodic Memories and Interpretations with Situated Scenario-based Ontological References (Baez Santamaria et al., MMSR 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.mmsr-1.6.pdf
Code
 cltl/EMISSOR
Data
ALFREDIEMOCAPMELDRECCON