Leveraging Multimodal Behavioral Analytics for Automated Job Interview Performance Assessment and Feedback

Anumeha Agrawal, Rosa Anil George, Selvan Sunitha Ravi, Sowmya Kamath S, Anand Kumar


Abstract
Behavioral cues play a significant part in human communication and cognitive perception. In most professional domains, employee recruitment policies are framed such that both professional skills and personality traits are adequately assessed. Hiring interviews are structured to evaluate expansively a potential employee’s suitability for the position - their professional qualifications, interpersonal skills, ability to perform in critical and stressful situations, in the presence of time and resource constraints, etc. Candidates, therefore, need to be aware of their positive and negative attributes and be mindful of behavioral cues that might have adverse effects on their success. We propose a multimodal analytical framework that analyzes the candidate in an interview scenario and provides feedback for predefined labels such as engagement, speaking rate, eye contact, etc. We perform a comprehensive analysis that includes the interviewee’s facial expressions, speech, and prosodic information, using the video, audio, and text transcripts obtained from the recorded interview. We use these multimodal data sources to construct a composite representation, which is used for training machine learning classifiers to predict the class labels. Such analysis is then used to provide constructive feedback to the interviewee for their behavioral cues and body language. Experimental validation showed that the proposed methodology achieved promising results.
Anthology ID:
2020.challengehml-1.6
Volume:
Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML)
Month:
July
Year:
2020
Address:
Seattle, USA
Venues:
ACL | Challenge-HML | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
46–54
Language:
URL:
https://aclanthology.org/2020.challengehml-1.6
DOI:
10.18653/v1/2020.challengehml-1.6
Bibkey:
Cite (ACL):
Anumeha Agrawal, Rosa Anil George, Selvan Sunitha Ravi, Sowmya Kamath S, and Anand Kumar. 2020. Leveraging Multimodal Behavioral Analytics for Automated Job Interview Performance Assessment and Feedback. In Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML), pages 46–54, Seattle, USA. Association for Computational Linguistics.
Cite (Informal):
Leveraging Multimodal Behavioral Analytics for Automated Job Interview Performance Assessment and Feedback (Agrawal et al., Challenge-HML 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.challengehml-1.6.pdf
Video:
 http://slideslive.com/38931260