Human-Robot Interaction (HRI) has evolved at a rapid rate in the last decade. This thesis tackles the aspect of contextualising the environment surrounding the robot in the form of semantics. A scene reconstruction system is proposed, composed of an offline stage (static objects) and an online one (dynamic objects). The system works as proof of concept. Its current application is to search for small objects of interest during the online stage, utilising larger objects found during the offline stage as landmarks. The semantics of the environment are computed by utilising a two-stage object segmentation pipeline, using YOLOv4 as an object detector and DeepLabV3+ for object segmentation. Object ontologies are used to create relations between static and dynamic objects. After offline and online reconstructions have been performed, the system projects the object masks to 3D coordinates. Many qualitative and quantitative results have been reported and demonstrate the robustness of the proposed system. The proof of concept is deemed successful since the system can utilise static objects as regions of interest to detect dynamic objects based on their ontological relations and infer where specific tasks will occur.