A master thesis from Aalborg University
Low Level Robot Control Using A Multi Modal Foundation Model: P10 project
[Low Level Robot Control Using A Multi Modal Foundation Model]
Author(s)
Term
4. semester
Education
Publication year
2024
Submitted on
2024-05-31
Pages
69 pages
Abstract
Robotic systems are often highly specialized, with little flexibility for different tasks. In this report, we outline our work on implementing our own robotic control stack in our pursuit to experiment on Octo, a multi-modal foundation model, for low-level control of a robotic manipulator. Octo is designed for flexibility, capable of running on various robotic hardware and performing a wide range of tasks. We fine-tuned Octo on our own data, recorded using tools developed for this project. This data is in a standardized format for future use in training robotic systems. To train and run Octo, we created a custom robot environment, integrated it with a Polymetis server wrapped in a ZeroRPC server, developed a VR control system for intuitive robot control, and built our own data recording tools. We modified existing Octo scripts to fit our use case, successfully fine-tuning and running Octo in our custom environment. Our model was trained to use two camera inputs and a task description to pick up an arbitrary object
Keywords
Documents
Colophon: This page is part of the AAU Student Projects portal, which is run by Aalborg University. Here, you can find and download publicly available bachelor's theses and master's projects from across the university dating from 2008 onwards. Student projects from before 2008 are available in printed form at Aalborg University Library.
If you have any questions about AAU Student Projects or the research registration, dissemination and analysis at Aalborg University, please feel free to contact the VBN team. You can also find more information in the AAU Student Projects FAQs.