Architectural Improvements for Real-Time Open-Vocabulary Semantic SLAM: Investigating possible improvements on segmentation and semantic embedding in stateof-the-art semantic SLAM methods
Translated title
Architectural Improvements for Real-Time Open-Vocabulary Semantic SLAM
Authors
Halasi, Tamás ; Jensen, Tue Burmeister
Term
4. semester
Education
Publication year
2026
Submitted on
2026-06-01
Abstract
This thesis explores how the architecture of modern online, open-vocabulary semantic SLAM systems can be improved so that robots can more accurately and efficiently understand and query 3D environments in real time using natural language. Building on the state-of-the-art OVO framework, RGB-D input is used to construct a semantic point cloud that remains continuously queryable and is later evaluated against ground-truth labels from the Replica dataset. The work focuses on two key components of the pipeline: instance segmentation and semantic embedding. First, the FindAnything oversegmentation algorithm is adapted and integrated into the OVO pipeline to isolate its contribution to overall performance and to study failure modes within a common SLAM backend. Second, a new instance segmentation pipeline, RADISH, is developed, combining a RADSeg encoder and YOLOE with RGB-D edge extraction to generate point prompts for SAM-based mask refinement, with a tighter coupling between segmentation and language-driven semantic descriptions. A visualization tool based on Rerun is also implemented to record and replay experiments run on AAU’s AI-LAB, supporting qualitative analysis and debugging. Experimental results with strict Replica labels show that the integrated FindAnything algorithm generally underperforms the original OVO baseline, largely due to instance ID misassignment and fragmentation under poor visibility. RADISH, while struggling with small objects, improves segmentation and semantic classification for medium-sized objects, and qualitative analysis indicates that many apparent misclassifications are semantically reasonable. When these ambiguities are accounted for through an enriched vocabulary, RADISH exhibits significant performance gains on medium and large objects. Overall, the thesis demonstrates the state-of-the-art potential of RADISH for real-time open-vocabulary semantic SLAM, while highlighting remaining challenges in small-object segmentation for resource-constrained, safety-critical applications such as search and rescue.
Dette speciale undersøger, hvordan arkitekturen i moderne, online, open-vocabulary semantisk SLAM kan forbedres, så robotter hurtigere og mere præcist kan forstå og forespørge 3D-miljøer med naturligt sprog i realtid. Udgangspunktet er OVO-rammeværket, som anvender RGB-D data til at opbygge en semantisk punkt-sky, der senere evalueres mod sandhedsdata fra Replica-datasættet. Projektet fokuserer på to centrale komponenter i pipelineen: instanssegmentering og semantiske embedding. Først integreres oversegmenteringsalgoritmen FindAnything i OVO for at isolere dens effekt på den samlede ydeevne og analysere fejlscenarier i samme SLAM-backend. Dernæst udvikles en ny instanssegmenteringspipeline, RADISH, der kombinerer en RADSeg-encoder, YOLOE samt RGB-D kantekstraktion til generering af punkt-prompts for SAM-baseret maskeforfining, med tæt kobling mellem segmentering og semantik via tekstbaserede beskrivelser. Til udvikling og kvalitativ evaluering implementeres desuden et visualiseringsværktøj baseret på Rerun, som muliggør afspilning af eksperimenter fra AAU’s AI-LAB. Eksperimenterne viser, at den integrerede FindAnything-algoritme generelt klarer sig dårligere end OVO’s oprindelige segmentering, bl.a. på grund af fejlagtige instans-ID’er og fragmentering under dårlig sigtbarhed. RADISH har vanskeligheder med små objekter, men forbedrer segmenteringen og semantisk klassifikation af mellemstore objekter, og en kvalitativ analyse viser, at flere tilsyneladende fejlklassifikationer er semantisk plausible. Når disse tages i betragtning via et udvidet ordforråd, opnår RADISH markante præstationsforbedringer for mellemstore og større objekter. Samlet demonstrerer arbejdet, at RADISH har potentiale som ny state-of-the-art løsning for realtids open-vocabulary semantisk SLAM, samtidig med at det tydeliggør de resterende udfordringer ved segmentering af små objekter i ressourcebegrænsede, sikkerhedskritiske applikationer som søg- og redningsoperationer.
[This abstract has been generated with the help of AI directly from the project full text]