Adaptive beam alignment and tracking in mmWave systems using Reinforcement Learning
Authors
Fisker, Peter Kjær ; Sand, Dennis Kjærsgaard
Term
4. term
Education
Publication year
2022
Submitted on
2022-06-02
Pages
173
Abstract
This thesis asks whether reinforcement learning (RL) is a practical way to keep highly directional radio beams aligned in millimeter‑wave (mmWave) communication between a single base station and a single user device. In mmWave systems, very high data rates rely on narrow beams; as the user moves or the environment changes, the beams must be continuously adjusted and tracked to maintain a reliable link. We build a realistic simulation of this setting: the radio channel is modeled with QuaDRiGa, and the user’s movement follows a mobility model that reflects real device motion. We first analyze the RL framework—both mathematically and in terms of how its states, actions, and rewards map to the beam alignment task. Based on this analysis, we implement two simple tabular RL solutions (which store learned values in tables rather than using neural networks): a centralized version with one agent and a distributed version with two cooperating agents. We tune the algorithm parameters under line‑of‑sight (LOS) conditions using broad sweeps over the parameter space. We then evaluate the tuned methods across multiple scenarios, including LOS and non‑line‑of‑sight (NLOS) links and environments with different noise levels. For comparison, we implement a straightforward heuristic baseline and test it under the same conditions. The results indicate that the simple two‑agent (multi‑agent) tabular RL performs best. It shows promising beam alignment and tracking in LOS, NLOS, and noisy scenarios, outperforming the heuristic reference in our tests. These findings suggest that RL is a viable approach to beam alignment and tracking in mmWave links under the studied conditions.
Denne afhandling undersøger, om reinforcement learning (RL) er en praktisk måde at holde meget retningsbestemte radiostråler justeret i millimeterbølge (mmWave) kommunikation mellem én basestation og ét brugerudstyr. I mmWave‑systemer afhænger de høje datarater af smalle stråler; når brugeren bevæger sig, eller omgivelserne ændrer sig, skal strålerne løbende justeres og spores for at bevare en stabil forbindelse. Vi opbygger en realistisk simulering: radiokanalen modelleres med QuaDRiGa, og brugerens bevægelse følger en mobilitetsmodel, der afspejler virkelig enhedsbevægelse. Først analyserer vi RL‑rammen – både matematisk og ved at kortlægge, hvordan tilstande, handlinger og belønninger passer til opgaven med strålejustering. Med udgangspunkt i denne analyse implementerer vi to enkle tabulære RL‑løsninger (som lagrer lærte værdier i tabeller frem for at bruge neurale netværk): en centraliseret version med én agent og en distribueret version med to samarbejdende agenter. Vi tuner metodeparametre under fri sigtelinje (LOS) ved hjælp af brede parametergennemsøgninger. Derefter evaluerer vi de tunede metoder i flere scenarier, herunder LOS og uden fri sigtelinje (NLOS) samt miljøer med forskellige støjniveauer. Som reference implementerer vi også en enkel heuristisk algoritme og tester den under de samme forhold. Resultaterne viser, at den simple to‑agent (multi‑agent) tabulære RL klarer sig bedst. Den giver lovende justering og sporing af stråler i både LOS, NLOS og støjende scenarier og overgår den heuristiske reference i vores tests. Disse resultater peger på, at RL er en anvendelig løsning til strålejustering og ‑sporing i mmWave‑forbindelser under de undersøgte betingelser.
[This apstract has been rewritten with the help of AI based on the project's original abstract]
Keywords