Automatic sonification of video sequences through object detection and physical modelling
Author
Term
4. Term
Education
Publication year
2017
Submitted on
2017-06-16
Abstract
The sounds generated by the objects that surround us is intrinsic to our life. We associate some specific sounds to the objects, its characteristics, and the actions that generate them, and we expect to hear the corresponding sound when we see that item. Similarly, one expect to see the corresponding object when we hear its sound. For example, on the street, when we hear a characteristic sound of a motor, we know that a car is approaching. There is, therefore, a particular relationship between the objects and the sound that they produce. In films, many of these sound effects are added in post-production, a method called "Foley". In this project, these sound effects will be generated automatically based mainly on one characteristic of the objects involved: their material. A system based on an object detector, an impact detector and a sound modeler will be presented. A perceptual evaluation in which the subjects will watch some videos and listen to the sounds predicted by the model will be performed. In this way, it will be tested if the sounds proposed by the model are played in accordance of what they expect to hear.
Documents