Solving Complex Problems with Deep Multi-Level Skill Hierarchies
Authors
Jacobsen, Tobias Lambek ; Abildgaard, Nicolaj Casanova
Term
4. term
Education
Publication year
2021
Submitted on
2021-06-04
Pages
9
Abstract
Reusing pre-trained skills is a common way to speed up Deep Reinforcement Learning (DRL) and support lifelong learning, where an agent continues learning across tasks. However, as environments include more tasks, the number of required skills and the size of the action space (the set of possible actions) grow, making training slower. We introduce Deep Multi-Level Skill Hierarchies (D-MuLSH), an extension of the Hierarchical Deep Reinforcement Learning Network (H-DRLN) that organizes skills across multiple levels. In D-MuLSH, simple skills are grouped into broader categories using pre-trained Major Skill Networks (MSN). This means the agent mainly learns when to use each category rather than choosing among every individual skill. In a game-like environment called ViZDoom, D-MuLSH reduced training time compared with H-DRLN.
Genbrug af forudtrænede færdigheder er en udbredt måde at gøre Deep Reinforcement Learning (DRL) hurtigere og støtte livslang læring, hvor en agent lærer på tværs af opgaver. Men når et miljø rummer flere opgaver, vokser antallet af nødvendige færdigheder og handlingsrummets størrelse (de mulige handlinger), hvilket gør træningen langsommere. Vi præsenterer Deep Multi-Level Skill Hierarchies (D-MuLSH), en udvidelse af Hierarchical Deep Reinforcement Learning Network (H-DRLN), som organiserer færdigheder i flere niveauer. I D-MuLSH grupperes simple færdigheder i bredere kategorier ved hjælp af forudtrænede Major Skill Networks (MSN). Dermed skal agenten primært lære, hvornår hver kategori skal bruges, i stedet for at vælge mellem alle enkeltfærdigheder. I et spil-lignende miljø kaldet ViZDoom reducerede D-MuLSH træningstiden sammenlignet med H-DRLN.
[This apstract has been rewritten with the help of AI based on the project's original abstract]