A master thesis from Aalborg University
Big Data Cloud Computing Infrastructure Framework: A framework for developing reproducible cloud computing infrastructures suitable for big data processing jobs
Author(s)
Term
4. Term
Publication year
2019
Submitted on
2019-09-14
Pages
98 pages
Abstract
Building an environment suitable to handle big data workloads involves using multiple software together to form the end result. We define a software framework as containing a suite of software packages to be used together to form a reproducible environment to run big data tasks on. Each choice of software is justified and its corresponding code is explained, as well as the resulting environment is demonstrated by running experimental big data tasks. The infrastructure is built by leveraging the Google Cloud Platform cloud computing provider to build the hardware. Terraform, an infrastructure manager, is used to communicate with the Google Cloud Platform API in order to programatically build the hardware infrastructure, while the Nix package manager is used to download, setup and configure the software packages. This framework can be used to build similar environments or adapt and further expand the code presented in this paper.
Building an environment suitable to handle big data workloads involves using multiple software together to form the end result. We define a software framework as containing a suite of software packages to be used together to form a reproducible environment to run big data tasks on. Each choice of software is justified and its corresponding code is explained, as well as the resulting environment is demonstrated by running experimental big data tasks. The infrastructure is built by leveraging the Google Cloud Platform cloud computing provider to build the hardware. Terraform, an infrastructure manager, is used to communicate with the Google Cloud Platform API in order to programatically build the hardware infrastructure, while the Nix package manager is used to download, setup and configure the software packages. This framework can be used to build similar environments or adapt and further expand the code presented in this paper.
Documents
Colophon: This page is part of the AAU Student Projects portal, which is run by Aalborg University. Here, you can find and download publicly available bachelor's theses and master's projects from across the university dating from 2008 onwards. Student projects from before 2008 are available in printed form at Aalborg University Library.
If you have any questions about AAU Student Projects or the research registration, dissemination and analysis at Aalborg University, please feel free to contact the VBN team. You can also find more information in the AAU Student Projects FAQs.