ReLight: Capturing spatial-temporal context in Road Traffic Signal Control using recurrency in POMDPs

Kjær, Peter ; Andersen, Samuel Alexander Vall

4. term

Software, Master

2022

13

Trafiktrængsel i byer belaster både miljø og økonomi. En måde at afhjælpe det på er bedre styring af trafiklys. Traditionelle systemer bruger faste, forudkodede cyklusplaner, som har svært ved at tilpasse sig komplekse og skiftende trafikmønstre. Forstærkningslæring (reinforcement learning, RL) kan styre dynamisk i realtid, men den almindelige antagelse (Markov-egenskaben) om, at den aktuelle tilstand rummer al nødvendig information, gør det svært at fange tidslige mønstre, når ikke alt i trafikken kan observeres på én gang. En del nyere arbejder kombinerer RL med forudsigelsesmoduler, men det kan introducere ekstra tab i træningen og svække generalisering. Vi introducerer Recurrent Light (ReLight), der behandler problemet som en delvist observerbar Markov-beslutningsproces (POMDP), hvor beslutninger bygger på historikken af tidligere "belief"-tilstande (skønnede, skjulte tilstande). Vi udnytter denne afhængighed ved at udvide et Deep Q-Network (DQN) med en LSTM (Long Short-Term Memory), et rekurrent neuralt netværk, som kan "huske" både kort- og langtidssignaler via skjulte tilstande, så rumlige og tidslige mønstre i trafikken fanges bedre. For at beskrive og lære signalcyklusser mere præcist foreslår vi desuden to prøvetagnings- og to træningsstrategier. I vores eksperimenter viser vi, at ReLight overgår state-of-the-art modeller på datasæt i flere skalaer, fra små til bydækkende.

Traffic congestion in cities harms both the environment and the economy. Optimizing traffic signals can help, but traditional systems rely on fixed timing plans that struggle with complex and changing traffic patterns. Reinforcement learning (RL) can adapt in real time, yet the usual Markov assumption—that the current snapshot contains all needed information—makes it hard to capture temporal patterns when the system is only partially observed. Some recent work adds prediction modules to RL, but these introduce extra training loss and can weaken generalization. We present Recurrent Light (ReLight), which frames the task as a Partially Observable Markov Decision Process (POMDP) where decisions depend on the history of past belief states (estimates of hidden conditions). We exploit this by adding an LSTM (Long Short-Term Memory) to a Deep Q-Network (DQN), giving the agent memory of short- and long-term signals through hidden states so it can capture spatial and temporal traffic patterns. To better represent and learn signal cycles, we also propose two data sampling and two training strategies. In experiments, ReLight outperforms state-of-the-art models across datasets at multiple scales, from small to city-wide.

[This summary has been rewritten with the help of AI based on the project's original abstract]

reinforcement learning

Download
View record in AAU Student Projects