The Spoon Is in the Sink: Assisting Visually Impaired People in the Kitchen

Katie Baker, Amit Parekh, Adrien Fabre, Angus Addlesee, Ruben Kruiper, Oliver Lemon


Abstract
Visual Question Answering (VQA) systems are increasingly adept at a variety of tasks, and this technology can be used to assist blind and partially sighted people. To do this, the system’s responses must not only be accurate, but usable. It is also vital for assistive technologies to be designed with a focus on: (1) privacy, as the camera may capture a user’s mail, medication bottles, or other sensitive information; (2) transparency, so that the system’s behaviour can be explained and trusted by users; and (3) controllability, to tailor the system for a particular domain or user group. We have therefore extended a conversational VQA framework, called Aye-saac, with these objectives in mind. Specifically, we gave Aye-saac the ability to answer visual questions in the kitchen, a particularly challenging area for visually impaired people. Our system can now answer questions about quantity, positioning, and system confidence in regards to 299 kitchen objects. Questions about the spatial relations between these objects are particularly helpful to visually impaired people, and our system output more usable answers than other state of the art end-to-end VQA systems.
Anthology ID:
2021.reinact-1.5
Volume:
Proceedings of the Reasoning and Interaction Conference (ReInAct 2021)
Month:
October
Year:
2021
Address:
Gothenburg, Sweden
Venue:
ReInAct
SIG:
SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
32–39
Language:
URL:
https://aclanthology.org/2021.reinact-1.5
DOI:
Bibkey:
Cite (ACL):
Katie Baker, Amit Parekh, Adrien Fabre, Angus Addlesee, Ruben Kruiper, and Oliver Lemon. 2021. The Spoon Is in the Sink: Assisting Visually Impaired People in the Kitchen. In Proceedings of the Reasoning and Interaction Conference (ReInAct 2021), pages 32–39, Gothenburg, Sweden. Association for Computational Linguistics.
Cite (Informal):
The Spoon Is in the Sink: Assisting Visually Impaired People in the Kitchen (Baker et al., ReInAct 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.reinact-1.5.pdf
Code
 aye-saac/aye-saac
Data
EQAGQAVisual Question AnsweringVizWiz