The COVID-19 pandemic made a significant impact on people's social and personal communication. Online meetups and video conferences have become a new normal. Although non-verbal behaviors and facial expressions play a crucial role under these circumstances, current online meeting platforms lack awareness in capturing and interpreting such rich information to facilitate a more desirable online communication experience.
Developing the current communicative capacity of online meeting experiences requires enriching functionalities and interfaces as human communication always accompanies postures, gestures, and facial expressions as a natural and intuitive oral language support. To mitigate this issue, visual support in non-verbal expressions involving rich and immediate feedback may provide improved online social presence and communication. I, therefore, designed an interactive tool, NEAS, to support non-verbal communication during online meetings using gesture and facial expression recognition through the lens of the human-centered machine learning perspective.
To achieve the research aim, first of all, I identified and addressed user pain points related to non-verbal expressions in online meetings through an online questionnaire. Then, 15 moments when users need non-verbal expressions in online ideation meetings and appropriate 13 non-verbal expressions for each moment were extracted through two design workshops. Based on this interactive design of NEAS, I collected behavioral data and built a model that recognizes important non-verbal expressive cues to visualize their meanings and implications through emojis. The interactive tool was demonstrated using posture & gestures and facial expressions recognition based on Ridge Classifier and Logistic Regression.
Through the demonstration, user evaluation was conducted in two approaches: quantitative(progress analysis, questionnaire) and qualitative approach(exit interview). Four insights were verified by user evaluation: 1) Participants were able to more actively and effectively express and quickly understand various non-verbal expressions through NEAS. 2) Implications for non-verbal expressions and visual aid that can be appropriately used in the actual online ideation meeting context. 3) In addition to building a robust machine learning system, it suggests that the user experience can be enhanced through user-centered improvement in system design. 4) Provide sufficient possibilities for participants to have more positive experiences and higher satisfaction in online ideation meeting progress through NEAS.
Therefore, this study demonstrated the necessity of designing interactive tools to aid non-verbal communication for online meetings. The results also suggested that this tool helps people better understand and use such expressions.