AI Sensing for Robotics using Deep Learning based Visual and Language Modeling

Yuvaram Singh, Kameshwar Rao JV


Abstract
An artificial intelligence(AI) system should be capable of processing the sensory inputs to extract both task-specific and general information about its environment. However, most of the existing algorithms extract only task specific information. In this work, an innovative approach to address the problem of processing visual sensory data is presented by utilizing convolutional neural network (CNN). It recognizes and represents the physical and semantic nature of the surrounding in both human readable and machine processable format. This work utilizes the image captioning model to capture the semantics of the input image and a modular design to generate a probability distribution for semantic topics. It gives any autonomous system the ability to process visual information in a human-like way and generates more insights which are hardly possible with a conventional algorithm. Here a model and data collection method are proposed.
Anthology ID:
2020.challengehml-1.8
Volume:
Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML)
Month:
July
Year:
2020
Address:
Seattle, USA
Editors:
Amir Zadeh, Louis-Philippe Morency, Paul Pu Liang, Soujanya Poria
Venue:
Challenge-HML
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
60–63
Language:
URL:
https://aclanthology.org/2020.challengehml-1.8
DOI:
10.18653/v1/2020.challengehml-1.8
Bibkey:
Cite (ACL):
Yuvaram Singh and Kameshwar Rao JV. 2020. AI Sensing for Robotics using Deep Learning based Visual and Language Modeling. In Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML), pages 60–63, Seattle, USA. Association for Computational Linguistics.
Cite (Informal):
AI Sensing for Robotics using Deep Learning based Visual and Language Modeling (Singh & JV, Challenge-HML 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.challengehml-1.8.pdf
Video:
 http://slideslive.com/38931257