Reinforcement Learning (RL) approaches are most often clueless of what to do in environments of only sparse extrinsic rewards. Nonetheless, humans and animals are able to learn under similar conditions due to curiosity, as it gives us an intrinsic drive to explore what is novel. In this work, we investigate the potential of how curiosity can enhance a model-based RL agent’s learning and encourage exploration in a visual environment of only sparse extrinsic rewards. We introduce a novel model-based curiosity-driven RL agent, that learns and uses a compact latent representation of the visual environment as input, employs the concept of Random Network Distillation (RND) to generate episodic intrinsic rewards and encourage curiosity-driven planning in a Monte Carlo Tree Search (MCTS). We demonstrate that curiosity enhances the learning of a model-based agent, as our proposed agent is able to solve a visual environment of sparse rewards, that is otherwise unsolvable by a model-free and model-based agent without curiosity. Finally, we examine the prospect of building a world model that can be used as an MCTS simulation environment.