Applying Facial Expression Analysis Software to Estimate the User Experience of LEGO building by Analyzing Facial Action Units

Abstract

Denne undersøgelse udforsker brugen af facial expression analysis (FEA) til at estimere brugeroplevelsen under LEGO-bygning ved at analysere facial action units (AU). Traditionel forskning af brugeroplevelser benytter ofte subjektiv selvrapportering, som kan være underlagt bias. Dette studie forsøger at anvende FEA til at opnå en mere objektiv og kvantificerbar tilgang. I undersøgelsen blev OpenFace 2.0, et værktøj som kan detektere såkalde \textit{facial landmarks} og genkende AUs, anvendt til FEA. Undersøgelsen involverede et remote setup med 18 deltagere, som byggede et LEGO-sæt derhjemme, mens deres ansigt blev optaget. Disse videoer blev behandlet for at udtrække AU-data, som derefter blev sammensat med deltagernes selvrapporterede vurderinger af fornøjelse, frustration, udfordring, kedsomhed og begejstring på en skala fra 1 til 7. På trods af en omfattende datindsamling og analyse resulterede de opstillede machine-learning-modeller, baseret på Random Forest regression og klassifikation, en ringe præstation i at forudsige de subjektive vurderinger ud fra AU-data. Dette kan skyldes, at undersøgelsen blev udført i enerum, hvilket kan have reduceret intensiteten i deltagernes ansigtsudtryk. Det kan ligeledes skyldes en utilstrækkelig variation i selvrapporteringer, som følge af at deltagerne ikke blev påvirket tilstrækkeligt i relation til de 5 parametre. Resultaterne antyder, at mens FEA har potentiale for anvendelse i forskning inden for brugervenlighed, er yderligere metodologiske forbedringer nødvendige. Fremtidig forskning bør overveje hyppigere selvrapporteringer, anvende mere komplekse machine-learning-modeller og potentielt inkorporere yderligere biometriske målinger for at opnå en forbedring i forudsigelser af emotionelle tilstande.

This study explores the application of Facial Expression Analysis (FEA) to estimate the user experience (UX) during LEGO building activities by analyzing Facial Action Units (AU). Traditional user experience (UX) research often relies on subjective self-reports, which are subject to biased. In contrast, we attempt to employ FEA to provide a more objective and quantifiable measure by capturing facial expressions. We utilized OpenFace 2.0, a tool for detecting facial landmarks and recognizing AUs, to analyze participants' facial expressions as they engaged in a LEGO building task. The study involved a remote experiment setup with 18 participants who built a LEGO set at home while being recorded on video. These videos were processed to extract AU data, which were then compared to participants' self-reported ratings of enjoyment, frustration, challenge, boredom, and excitement on a 7-point scale. Despite extensive data collection and analysis, the machine learning models trained, based on Random Forest regression and classification, to predict subjective assessments from AU data, showed poor performance. This may be due to the solitary nature of the task affecting participants' facial expressiveness or insufficient variation in self-reported experiences. The findings suggest that while FEA holds potential for UX research, further methodological refinements are needed. Future research should consider more frequent self-assessments, deploying more complex machine learning models, and potentially incorporating additional biometric measures to improve accuracy of emotional state predictions.

A master thesis from Aalborg University

Applying Facial Expression Analysis Software to Estimate the User Experience of LEGO building by Analyzing Facial Action Units