Performance Evaluation of Crowdsourced HCI User Studies
Authors
Christensen, Allan Kærgaard ; Pedersen, Simon Andre'
Term
4. term
Education
Publication year
2015
Submitted on
2015-05-27
Pages
77
Abstract
Dette speciale undersøger, om crowdsourcing—rekruttering af deltagere online—kan fungere som deltagergrundlag i brugerundersøgelser inden for menneske–computer-interaktion (HCI). Målet er at øge deltagerantallet og mindske afhængigheden af såkaldte WEIRD-populationer (vestlige, veluddannede, industrialiserede, rige og demokratiske). Først gennemførte vi et laboratoriebaseret pilotforsøg, hvor vi sammenlignede to måder at styre et spil på: berøring og tilt (at vippe enheden). Berøring klarede sig bedre end tilt. Derefter sammenlignede vi tre grupper på tværs af to opgaver: laboratoriedeltagere, en informeret crowd (som vidste, at de deltog i et eksperiment), og en uinformeret crowd. Den første opgave var et spilniveau styret med berøring. Den anden var en måludvælgelsesopgave baseret på Fitts’ lov, en model der beskriver, hvordan sværhedsgrad afhænger af målets størrelse og afstand; dens sværhedsindeks (ID) stiger, når mål bliver mindre eller længere væk, og vi brugte det til at estimere det mindste mål, der kan vælges uden større anstrengelse. Resultaterne viste, at laboratoriegruppen i spilopgaven overgik begge crowds, og at den informerede crowd klarede sig bedre end den uinformerede. I Fitts’-opgaven lavede laboratoriegruppen færre fejl samlet set og viste en signifikant stigning i fejl mellem ID 3.70 og 4.64. Den informerede crowd havde en markant stigning i fejl mellem ID 2.81 og 3.70. Den uinformerede crowd lavede generelt for mange fejl til at identificere en tydelig tærskel. På tværs af alle grupper lå det mindste valgbare mål på touch-enheder mellem 2 mm og 4 mm. Samlet peger resultaterne på, at crowdsourcing kan udvide rækkevidden, men ikke matchede laboratoriets præstation i disse opgaver; at det hjælper at informere online-deltagere; at berøring er mere pålidelig end tilt til styring; og at resultaterne giver et praktisk interval for mindste målstørrelse på touch.
This thesis examines whether crowdsourcing—recruiting participants online—can serve as a viable participant pool for Human–Computer Interaction (HCI) user studies. The aim is to increase participation and reduce reliance on WEIRD populations (Western, Educated, Industrialized, Rich, and Democratic). We first ran a lab-based pilot comparing two ways to control a game: touch input and tilt (device rotation). Touch outperformed tilt. We then compared three groups across two tasks: lab participants, an informed crowd (told they were in an experiment), and an uninformed crowd. The first task was a touch-controlled game level. The second was a target-selection task based on Fitts’ law, a model that relates movement difficulty to target size and distance; its index of difficulty (ID) increases as targets get smaller or farther, and we used it to estimate the smallest target that can be selected with little effort. Results showed that, in the game level, the lab group outperformed both crowds, and the informed crowd did better than the uninformed crowd. In the Fitts’ law task, the lab group made fewer errors overall and showed a significant increase in errors between IDs 3.70 and 4.64. The informed crowd showed a spike in errors between IDs 2.81 and 3.70. The uninformed crowd made too many errors overall to identify a clear threshold. Across all groups, the smallest selectable target on touch devices fell between 2 mm and 4 mm. Taken together, the findings suggest that crowdsourcing can broaden reach but did not match lab performance in these tasks; informing online participants improved data quality; touch is more reliable than tilt for control; and the results provide a practical range for minimum touch target sizes.
[This abstract was generated with the help of AI]