Accelerated Data Processing Saves Time, Allows for More Thorough Research

How co-located data projected onto multiple Virtual Machines on ADAPT radically saved time, effort, and resources for ABoVE’s study on the impact of climate change on our continent’s water supply.
 
Project Goal:
A key aspect of any ecosystem is surface water, yet for the Arctic-Boreal Vulnerability Experiment (ABoVE) study region (pictured below), water was one of the least characterized features. Therefore, ABoVE researchers set out to build comprehensive surface water maps of the vast region, which required processing and analyzing more than 100,000 Landsat satellite scenes.

 
This map depicts the study domain of this project. The region outlined in red was the original region ABoVE planned to document. With help from ADAPT, researchers were able to expand  their analysis to the region outlined in purple due to the amount of time saved.


Challenge:
The original data processing plan would have taken 9 to 12 months, as researchers would have had to order Landsat data, download a few terabytes to personal computers, process it, delete it, and start over again until all of the data was processed. Because this process was so time consuming, reprocessing was left out of the original plan.

Solution:
Fortunately, the NASA Center for Climate Simulation (NCCS) was prototyping a virtual system, initially referred to as the Science Cloud, which served the computing needs of the ABoVE team. This system has since evolved—mainly due to the work done on this project—and is named the Advanced Data Analytics Platform (ADAPT). By downloading and storing all Landsat data in ADAPT, the process was significantly accelerated. Furthermore, by corralling up to 48 of ADAPT’s Virtual Machines (VMs) for parallel data processing, researchers were able to process the Landsat scenes much faster and produce a seamless surface water map—all in a mere 6 weeks. Below is a diagram showing the performance of each processing plan:


 


Computing System:
ADAPT is a computing system at the NCCS that combines high performance computing and virtualization technologies to create an on-site private cloud. Its customizable operating environments enable researchers to process and analyze large datasets using the analytics solution of their choice. ADAPT also houses multiple data repositories that can be accessed for research.

Value Added:
“[The] speed-up allowed us to expand the geographic area to most of Canada,” said Mark Carroll, a Senior Support Scientist at NASA Goddard Space Flight Center. Working with ADAPT gave the ABoVE team the time and resources they needed to expand the scope of their project and analyze more data in order to understand the impact of climate change on our continent’s water supply. Below is a graph depicting the processing time difference between the two plans:
 
 

Also as a result of this project, the Landsat data now remains available to all ADAPT users for scientific processing. For more information regarding this story, visit Accelerating Water Map-Making for the ABoVE Field Campaign