Universal Speech Enhancement: A Speech Language Modelling Perspective
Author
Kühne, Nikolai Lund
Term
4. semester
Education
Publication year
2026
Abstract
This thesis explores how modern speech language models can be used for universal speech enhancement, where the goal is to remove many different types of distortion from audio signals and recover speech that is as clean as possible. Universal speech enhancement is defined broadly to include tasks such as denoising, dereverberation, declipping, bandwidth extension, acoustic echo cancellation, target speaker extraction, and packet loss concealment. The work focuses on three main design choices in such systems: the choice of modelling framework (non-autoregressive, autoregressive, and masked generative modelling), the choice of neural audio codec, and the choice of self-supervised speech representation model. Based on transformer architectures and neural audio codecs, several model variants are constructed and trained to enhance distorted speech signals, and then evaluated on languages and distortions that were not seen during training. Their performance is compared with state-of-the-art baselines from the literature using perceptual quality metrics such as DNSMOS, UTMOS, and SCOREQ. The results show that autoregressive modelling consistently outperforms non-autoregressive modelling for speech enhancement across all tested benchmarks, while the choice among different self-supervised speech representation models has only a minor impact on performance. In contrast, the choice of neural audio codec is critical: using a codec with very high reconstruction quality and a large codebook can make language modelling more complex and actually reduce enhancement performance. The thesis discusses the practical implications of these design choices, the challenges of evaluating language-model-based speech enhancement, and outlines how future systems can be made more robust and useful in applications such as hearing assistive devices and speech-driven interactive technologies.
Denne afhandling undersøger, hvordan moderne sproglige modeller for tale kan bruges til universel taleforbedring, hvor målet er at fjerne mange forskellige typer forvrængning fra lydsignaler og genskabe så ren tale som muligt. Universel taleforbedring omfatter en række delopgaver såsom støjreduktion, dereverberation, declipping, båndbreddeudvidelse, ekkofjernelse, udtræk af måltaleren og kompensation for pakkeforsvind i kommunikationssystemer. Afhandlingen fokuserer på tre overordnede designvalg i sådanne systemer: valg af modelleringsramme (ikke‑autoregresse, autoregresse og masked generative modelling), valg af neuralt lydkodek og valg af selv-superviseret tale-repræsentationsmodel. Med udgangspunkt i transformer-baserede sproglige modeller og neurale audio-kodeks konstrueres og evalueres flere varianter, der trænes til at rense tale signaler med forskellige typer forvrængning. Modellerne testes på sprog og forvrængninger, der ikke er set under træning, og sammenlignes med etablerede metoder fra litteraturen ved hjælp af perceptuelle kvalitetsmål som DNSMOS, UTMOS og SCOREQ. Resultaterne viser, at autoregressiv modellering konsekvent giver bedre taleforbedring end ikke-autoregressiv modellering på tværs af alle undersøgte benchmarks, mens valget mellem forskellige selv-superviserede tale-repræsentationer kun påvirker ydeevnen svagt. Derimod har valget af neuralt lydkodek stor betydning: et kodek med meget høj rekonstruktionskvalitet og stor kodebog kan gøre sproglig modellering mere kompleks og dermed forringe den praktiske taleforbedringsydelse. Afhandlingen diskuterer de metodiske konsekvenser af disse designvalg, udfordringer ved at evaluere sproglige modeller for taleforbedring og peger på, hvordan fremtidige systemer kan designes mere robuste og anvendelige, eksempelvis til høreteknologiske hjælpemidler og talebaserede interaktive systemer.
[This abstract has been generated with the help of AI directly from the project full text]