Toward Gaze-Guided Prosthesis Preshaping: A Transferable Pipeline for Data-Driven Prosthesis Grasp Prediction and the Limits of Single-Frame Grasp Inference
Author
Dalgaard, Hans Henrik
Term
4. semester
Education
Publication year
2026
Submitted on
2026-06-03
Abstract
Many people who receive an upper-limb prosthesis eventually stop using it. A key reason is that current control methods are often slow and hard to use. The most common approach, myoelectric control (using muscle signals to control the device), struggles to reliably control several movement degrees of freedom at the same time. This has motivated the exploration of shared-control systems, where part of the effort of understanding the user’s intention is shifted to other sensors, such as tracking where the user looks (gaze). This thesis introduces a modular, automated annotation pipeline that works without manual labels and converts raw egocentric multimodal recordings (captured from the user’s point of view) into data describing how the hand is pre-shaped before a grasp. The pipeline performs three main tasks: it detects gaze fixations on objects in the scene, detects and labels grasp events based on hand–object contact, and links earlier gaze fixations to the grasps they appear to anticipate. The method was applied to 62 sequences from the HOT3D dataset, producing 952 linked grasp events and 9,982 fixation samples. The resulting annotations were evaluated along three dimensions. A spatial analysis shows that, for objects that afford a clear, specific grasp type, the user’s anticipatory gaze tends to land on the part of the object where the functional grasp is actually made. For compact objects that can be grasped over their entire surface, this alignment between gaze and grasp region is much weaker. In addition, the individual modules of the pipeline were evaluated both quantitatively and qualitatively, and each module showed strong performance on its specific task. A further “downstream” evaluation used graph neural networks to predict MANO hand-pose principal components (from a standard 3D hand model) from a gaze-annotated 3D object mesh. These models were able to recover the dominant pose component (with R² ≈ 0.15–0.20), whereas classical regression methods failed completely on this task. However, an ablation study (systematically removing input channels) showed that adding gaze information did not improve prediction performance for any of the tested neural network architectures. Overall, the results indicate that the pipeline produces grasp data with meaningful, geometry-related structure. At the same time, they suggest that anticipatory gaze needs to be integrated in a more sophisticated way than a simple per-vertex gaze feature if it is to be fully exploited for prosthesis control.
Mange mennesker, der får en armprotese, ender med ikke at bruge den. En vigtig grund er, at styringen ofte er langsom og svær at forstå. Den mest udbredte metode, myoelektrisk kontrol (styring via muskelsignaler), har svært ved at styre flere bevægelsesfrihedsgrader samtidigt på en stabil måde. Derfor undersøges såkaldte shared-control-systemer, hvor noget af fortolkningen af brugerens intentioner flyttes over til andre typer sensorer, for eksempel øjenblikretning (gaze). Denne afhandling præsenterer en modulopbygget, automatiseret annoteringspipeline, der uden manuelle etiketter omdanner rå, egocentriske multimodale optagelser (set fra brugerens perspektiv) til data om forberedelse af greb. Pipenlinen gør tre ting: den finder øjenfikationer på objekter i scenen, den opdager og mærker selve grebhændelserne ud fra kontakt mellem hånd og objekt, og den forbinder tidligere øjenfikationer med de greb, de forudsiger. Metoden blev anvendt på 62 sekvenser fra HOT3D-datasættet og resulterede i 952 sammenkædede grebhændelser og 9.982 fikationsprøver. Annoteringerne blev vurderet på tre måder. En rumlig analyse viser, at for objekter, der indbyder til en bestemt type greb, rammer den forudgående blikretning den del af objektet, hvor grebet faktisk udføres. For kompakte objekter, der kan gribes over hele overfladen, er denne sammenhæng derimod svagere. Derudover gennemføres en kvantitativ og kvalitativ evaluering af de enkelte moduler i pipelinen, som viser, at de hver især præsterer godt. Endelig foretages en “downstream” evaluering: her bruges grafneuronale netværk til at forudsige hovedkomponenter af MANO-håndpositurer (en standardmodel for håndens 3D-form) ud fra et 3D-objektmesh, der er beriget med gaze-annoteringer. Disse modeller kan genskabe den dominerende posekomponent (forklaringsgrad R² ≈ 0,15–0,20), mens klassiske regressionsmetoder slet ikke lykkes med denne opgave. En ablation (kontrolleret frakobling af informationskanaler) viser dog, at selve gazekanalen ikke forbedrer forudsigelserne for nogen af de afprøvede neurale netværksarkitekturer. Samlet set tyder resultaterne på, at pipelinen genererer grebsdata med reel, geometrisk forankret struktur, men at den forudgående blikretning skal integreres mere avanceret end blot som en simpel per-vertex gaze-funktion for at kunne udnyttes fuldt ud i styring af proteser.
[This abstract has been rewritten with the help of AI based on the project's original abstract]
Keywords