Dynamic Resource Allocation Using Virtual Machines and Parallel Data Processing in the Cloud

The main enabling technology for cloud computing is virtualization which generalize the physical infrastructure and makes it easy to use and manage. Virtualization is used to allocate resources based on their needs and also supports green computing concept. Parallel data processing has emerged to be one of the killer applications for Infrastructure-as-a-Service (IaaS) clouds. The processing frameworks which are currently used have been designed for static, homogeneous cluster setups and disregard the particular nature of a cloud. In this paper, the authors are applying the concept of "skewness" to measure the unevenness in the multi-dimensional resource utilization of a server. By minimizing skewness, they can combine different types of workloads and improve the overall utilization of server resources and discuss the opportunities and challenges for efficient parallel data processing in clouds using "Nephelle's Architecture". Nephel's architecture offers efficient parallel data processing in clouds. It is the first data processing framework for the dynamic resource allocation offered by today's IaaS clouds for both, task scheduling and execution.