How an xAI-first Approach can Misguide User-Centred Design: An Anaesthetic Preoperative Case Study
Authors
Hollensberg, Linus Nørgaard ; Christiansen, Frederik Roland
Term
4. term
Education
Publication year
2026
Abstract
This thesis explores how a narrow focus on explainable artificial intelligence (xAI) can conflict with user-centred design when developing clinical decision support systems for pre-anaesthetic assessments. The starting point is that pre-anaesthetic assessments of the airway and patient records are time-consuming yet critical for preventing serious complications during anaesthesia. Modern machine learning models and large language models can assist by analysing multimodal data and electronic health records, but their increasing complexity raises challenges of transparency, interpretability, and trust. Explainable AI aims to address these issues through techniques such as saliency maps, feature attribution, and other forms of explanation, yet prior research shows that even technically sound explanations may increase cognitive load and lead to under- or overtrust if they do not fit clinicians’ real-world workflows. The thesis addresses this research and practical challenge through a user-centred, co-design-based development of an AI-driven clinical decision support system for pre-anaesthetic assessment. An initial round of interviews is conducted to elicit user requirements and understand existing work practices, followed by a visual review of the state of the art in CDSS and xAI interfaces. Based on these insights, 30 interface designs are created and evaluated in a co-design workshop with three anaesthesiologists, resulting in 36 concrete design criteria for a prototype. These criteria inform an AI-based text summarisation dashboard, which is subsequently evaluated with four anaesthesiologists. Although the goal was to create a more trustworthy and understandable xAI interface, the final evaluation shows that clinicians reject the prototype because it does not adequately support their current workflow. The thesis therefore concludes that an xAI-first approach can misguide the design process and argues that developers should start from clinicians’ core tasks and workflows and then adapt explainability features to those, rather than optimising explainability in isolation.
Denne afhandling undersøger, hvordan et ensidigt fokus på forklarlig kunstig intelligens (xAI) kan komme i konflikt med brugercentreret design, når der udvikles kliniske beslutningsstøttesystemer til præ-anaæstetiske vurderinger. Udgangspunktet er, at præ-anaæstetiske vurderinger af luftveje og patientjournaler er tidskrævende, men samtidig kritiske for at forebygge alvorlige komplikationer ved anæstesi. Moderne maskinlæringsmodeller og store sprogmodeller kan hjælpe med at analysere multimodale data og elektroniske patientjournaler, men deres kompleksitet skaber udfordringer med gennemsigtighed, fortolkbarhed og tillid. Forklarlig AI sigter mod at afhjælpe dette gennem teknikker som saliency maps, feature-attribution og andre forklaringstyper, men forskning peger på, at selv teknisk gode forklaringer kan øge kognitiv belastning og føre til enten under- eller overdrevet tillid, hvis de ikke passer til klinikeres faktiske arbejdsprocesser. Afhandlingen adresserer denne forskningsmæssige og praktiske udfordring gennem en brugercentreret, co-design-baseret udvikling af et AI-baseret CDSS til præ-anaæstetiske vurderinger. Først gennemføres et indledende interviewforløb for at kortlægge krav til systemet og forstå eksisterende arbejdsgange, efterfulgt af en visuel analyse af state-of-the-art inden for CDSS- og xAI-grænseflader. På baggrund heraf udvikles 30 forskellige interface-designs, som evalueres i en co-design-workshop med tre anæstesiologer, hvilket resulterer i 36 konkrete designkriterier til en prototype. Disse kriterier omsættes til et AI-baseret tekstopsummeringsdashboard, der efterfølgende testes med fire anæstesiologer. Selvom målet var at skabe et mere betroet og forståeligt xAI-interface, viser den afsluttende evaluering, at klinikerne afviser prototypen, fordi den ikke understøtter deres eksisterende arbejdsgang. Afhandlingen konkluderer derfor, at en xAI-first tilgang kan mislede designprocessen, og argumenterer for, at udviklere bør tage udgangspunkt i klinikernes kerneopgaver og workflow og herefter tilpasse forklarlighedsfunktionerne til disse, frem for at optimere forklarligheden isoleret.
[This abstract has been generated with the help of AI directly from the project full text]