Models defining the motion of objects in images are an important field in Computer Vision. A common drawback of many models it that sets of trajectories with different motions but sharing some common paths are described individually, therefore producing redundant and uncorrelated data. These common paths can be described as action primitives, and models linking action primitives are necessary to correlate them. This thesis defines a framework to track objects visually using color segmentation and Hidden Markov Models, record motion trajectories, identify action primitives and build a single model from different trajectories describing them jointly and efficiently. The action primitives model can be used as a learning model for a robot with a higher level definition of the actions performed. The framework is written as a C++ programming library and a complete implementation is provided which fulfills all the requirements for object detection, tracking, motion recording and model building.