Leveraging Commercial Clouds As Adjunct To On-Premises HPC Cluster Resources

Paulo Paz

Abstract

Cloud Computing has long been used to supplement on-premises resources. More recently Cloud Bursting has become a relatively new technology that is being made available to the HighPerformance Computing Community. Its appeal lies in its ability to either run jobs in the cloud when demand for on-premises resources have exceeded their capacity or pay on-demand for computing power without the costs associated with setting up the infrastructure.

Recently the NCCS worked on a project involving Cloud Bursting which was used as a motivation behind this study. NCCS’s goal was to use satellite imagery data for counting Introduction trees and shrubs to estimate biomass and carbon uptake in the Sub-Saharan African region. The initial workflow burst onto Amazon Web Services (AWS) ; orthorectified mosaicking shown right, removed the effect of image perspective and terrain on the images and then stitched the images together [1]. The following processing workflow then carried out the actual algorithm for counting. Using AWS resources for Cloud Bursting, the NCCS was able to reduce their estimate of 10 months for dedicated on premises resources to approximately 1 month.

As a result our objective for this project is two-fold; establish a suite of benchmarks to:

  1. Verify that the computing power of a Public Cloud is comparable to that of on-premises resources.
  2. Using software from Adaptive Computing Enterprises Inc. and from Microsoft (previously Cloud Computing) demonstrate how to configure a Cloud Bursting environment, and study how responsive a Public Cloud Provider is to elastic computing.