Testing, Testing - A comparative study of usability evaluation methods for mobile systems

Hansen, Lars Michael ; Bager, Andreas

4. term (INF10 - Master Thesis)

Informatics, Master

2007

2012-02-14

The work presented in this thesis serves to shed some light on the current practices of the HCI research community regarding how and why researchers evaluate the way they do. Furthermore, a comparative study of the methods when applied to a mobile system, serves to outline the strengths and weaknesses of the most commonly used usability evaluation methods. A literature review was conducted, in order to identify how and why professionals in the field of Human-Computer Interaction evaluate mobile systems as they do. Based on the results from the review it was concluded that researchers typically conduct usability evaluations of mobile systems by expert evaluations, laboratory evaluations, field evaluation or longitudinal evaluations. The first two methods are typically conducted without including the context of use while the latter includes the context of use. Based on the knowledge provided by the literature review, a set of evaluation sessions were arranged, utilizing the four most commonly applied evaluation methods. By doing so the necessary first-hand knowledge was obtained and an analysis of the individual methods paved the way for a thorough comparison of the characteristics of the evaluation methods. It was eventually concluded that all methods possess different strengths and weaknesses and that no superior method could be identified. Thus in thesis researchers and practitioners will find not find an unambiguous answer to what single method they should apply. Stimulated by the analysis of the applied methods, a new method was constructed in order to investigate if and how the strengths of different methods could be combined in order to overcome their weaknesses. The novel method was inspired by cultural probes and based on the experiences gained with the longitudinal and contextual evaluations. The resulting method – the video probe evaluation – was applied, evaluated and compared to the original four. In the end it was concluded that by overcoming known obstacles by combining known techniques, new obstacles arose. Thus the novel approach suffered the same sentence as the original four and was labeled as a valuable but not superior method.

Download
View record in AAU Student Projects