NASA’s SMCE Team Helps Scientists Develop Collaborative Cloud Computing Projects

NASA’s SMCE Team Helps Scientists
Develop Collaborative
Cloud Computing Projects

*CISTO Chief and SMCE Government Manager Dr. Daniel Duffy presenting in front of the NASA hyperwall during Supercomputing 2022 in Dallas, Texas. James Kling/NASA Photo.*

*The animated GIF shows the unique (and shared) perspectives of the SMCE team on the SMCE and its capabilities. Source: NASA.*

Cloud computing is a concept that refers to compute, storage, and network infrastructure assembled as a platform on which applications and systems can be deployed quickly and scaled up or down as needed by users on the fly. It is a self-service system in which users can select their needed resources from a dashboard and quickly run applications in the assembled environment.

Although NASA has significant on-premises computing and data capabilities located at NASA centers such as NASA Goddard Space Flight Center in Greenbelt, Maryland, and NASA Ames Research Center in Mountain View, California, commercial cloud computing has significant advantages for NASA.  

Over the past decade or more, commercial cloud computing has matured considerably, becoming a viable platform for scientific computing at NASA and beyond. To take advantage of this rapidly growing capability, the Computational and Information Sciences and Technology Office (CISTO) at NASA Goddard developed the Science Managed Cloud Environment (SMCE) in spring 2020, initially using Amazon Web Services (AWS).

The SMCE project evolved from an earlier cloud project led in 2015 by NASA Senior Engineer and SMCE Advisor Mike Little at the Advanced Information Systems Technology (AIST) program within NASA’s Earth Science Technology Office (ESTO). ESTO identifies, develops, and demonstrates advanced information system technologies to meet emerging requirements across NASA’s Earth Science Division (ESD).

The SMCE project addressed security and infrastructure challenges that scientists were facing. NASA is a highly collaborative scientific research organization that works closely with government agencies, universities, industry, international partners, and the public. Trying to create a virtual and collaborative research environment for such a diverse set of stakeholders with their unique array of IT security requirements is extremely difficult to navigate in a traditional (on-premises) computing environment.

Although the majority of NASA science research requires the lowest level of IT security governance and controls (i.e., FISMA low), all on-premises NASA compute capabilities require a much higher level of control, a process which results in barriers to enabling the level of collaboration needed. The goal of the SMCE project is to significantly decrease these barriers and accelerate collaborative research and development while still allowing most of the control to be decided by the scientists involved in the project.

To leverage commercial cloud computing for science, the SMCE team created a managed cloud environment that provides researchers with a standard set of secure machine instances to use, along with centralized security and monitoring, support for cloud-native compute and storage solutions specifically engineered for science, as well as tools for budgeting and financial management. The SMCE-managed ecosystem allows scientists to focus on science.

Through close collaboration with its science partners, SMCE engineered a cloud-native scientific software environment using a combination of JupyterHub and parallel computing in an easily deployable ecosystem with common tools used for scientific research and analysis. Users log directly into the cloud using a dashboard to obtain access to a JupyterLab Notebook, which provides direct access to data and parallel computing capabilities. This provides a familiar and intuitive environment in which researchers can quickly start to be productive without having to become cloud experts. 

As word spread about the ease of use and value of the SMCE's platform, it became increasingly popular and is now utilized by 30 DaskHubs and 80 unique projects within NASA. The SMCE is currently growing approximately 50 percent per year, supporting a wide range of projects that cut across all of the NASA Science Mission Directorate. Four notable examples of SMCE projects include:

The NASA Earth Information System. The SMCE team was approached in late 2020 by a group of NASA Earth scientists with a concept for a unified Earth Information System (EIS) that would bring together numerical models, satellite observations, and other resources onto a common computing platform, allowing internal and external stakeholders to work collaboratively. In just a few short months, after an extensive co-design process, the SMCE team delivered a robust platform utilizing Jupyter notebooks, Dask for parallel processing, and traditional high-performance computing in a single, low-cost, cloud-based environment.

NASA’s EIS projects are designed to provide collaborative environments in which a diversity of data can be brought together to better understand Earth’s fires, freshwater, sea level, carbon, and other aspects.

Impact: When researchers can bring both observational and model data together into one platform, they can more quickly derive an understanding of Earth systems.

*A screen shot of the homepage and dashboard of the NASA Earth Information System, a pilot project supported by the SMCE.*

The NASA Community Coordinated Modeling Center (CCMC). A multi-agency partnership to develop the next generation of space science and space weather applications.

Impact: The SMCE enables the CCMC and its partners to jointly develop, test, and verify codes in a collaborative environment before moving into operations. 

The Harmonized LandSat Sentinel-2 (HLS) dataset. Landsat and Sentinel-2 data represent the most widely accessible moderate-to-high spatial resolution multispectral satellite measurement. Following the launch of the two Sentinel-2 satellites in 2015 and 2017, the potential for synergistic use of Landsat and Sentinel-2 data created unprecedented opportunities for timely and accurate observation of Earth status and dynamics. Harmonization of the Landsat and Sentinel-2 data is of paramount importance for the scientific community.

Researchers at NASA Goddard accessed a combination of satellite data in the cloud to create the harmonized HLS dataset.

Impact: The SMCE managed cloud environment enabled collaborators to gain quick access to data and process the data on over 30% of the Earth globe before handing the algorithm off to NASA operations to run it in production mode. 

Above: The European Space Agency (ESA) Sentinel-2 satellite, a constellation comprised of two polar-orbiting satellites placed in the same sun-synchronous orbit, phased at 180° to each other. Sentinel-2 monitors Earth’s land surface using a wide swath (290 kilometers wide) and a high revisit time (10 days at the equator with one satellite, and 5 days with 2 satellites under cloud-free conditions, which results in 2–3 days at mid-latitudes) will support monitoring of Earth's surface changes over time. Artist rendering by ESA.

Above: The Landsat 8 satellite, a collaboration between NASA and the U.S. Geological Survey (USGS). Its payload consists of two science instruments —the Operational Land Imager (OLI) and the Thermal Infrared Sensor (TIRS). These two sensors provide seasonal coverage of the global landmass at a spatial resolution of 30 meters (visible, NIR, SWIR); 100 meters (thermal); and 15 meters (panchromatic). Artist rendering by NASA.

HelioCloud. A collaborative science environment that is designed with tools specific to heliophysics with an emphasis on comparing different datasets, sharing code, training, and much more. The HelioCloud held an international training course leveraging the SMCE that reached 400 people, including participants in Europe and Australia.

Impact: By bringing data and applications into the classroom, learners benefit from a rich and agile training environment, regardless of their experience. 

*The 2020 image above is a combination of images obtained by NASA's Solar Dynamics Observatory (SDO) over the prior 10 years. Credit: NASA's Goddard Space Flight Center.*

The SMCE is also used for a wide range of training for its customers in different fields, including biological sciences, artificial intelligence, machine learning, and Python coding. These projects would be extremely difficult to achieve using traditional approaches to information technology and on-premises resources. The SMCE enables these types of research, collaboration, and training in a fraction of the time it would otherwise take. 

SMCE Advisor and NASA Senior Engineer Mike Little observed, “SMCE’s design is unique in several ways. It enables both NASA and non-NASA participants in a research project to use the same data and computing environment, which enhances collaboration and enables faster time to discovery. Beyond these scientific advantages, SMCE’s security posture is managed by the SMCE team so that it is safe and secure for science investigations, not placing any other NASA asset at risk.”

Two early adopters of the SMCE at NASA also weighed in about the impact of the SMCE platform to their specific NASA projects. The first comments were provided by astrophysicist Tess Jaffe, Chief Archive Scientist at NASA Goddard Space Flight Center’s High Energy Astrophysics Science Archive Research Center (HEASARC) – the primary archive for NASA and other space agencies' missions studying electromagnetic radiation from extremely energetic cosmic phenomena.

“We have benefited from the expertise of the SMCE team and the infrastructure they built,” Jaffe said. “Working with the SMCE has enabled a collaboration among NASA's astrophysics archives (including those run independently of NASA) that would otherwise not have been feasible. Our project, a science platform for the NASA astrophysics community, is intended to be as open as possible, using only publicly available data and software: this is precisely what the SMCE was designed for. We are also using the SMCE platform to collaborate with other projects with common needs. The SMCE is exactly what we all need to enable open science.”

*Astrophysicist and HEASARC Chief Archive Scientist Tess Jaffe. NASA photo.*

The Global Modeling and Assimilation Office (GMAO) at NASA Goddard is another group that leverages the SMCE platform to support NASA's Earth Science mission. The GMAO’s Earth system modeling and data assimilation maximizes the impact of satellite observations of the atmosphere, ocean, land, and cryosphere through analyses and predictions. The GMAO’s Goddard Earth Observing System (GEOS) is a group of model components that can be connected in a flexible manner in order to address questions related to different aspects of Earth Science.

GMAO Atmospheric Scientist Christoph Keller is an enthusiastic early adopter of the SMCE. "Using the SMCE platform has greatly simplified the analysis of model output produced by the NASA GEOS composition forecast system," emphasized Keller. "It has enabled us to easily compare air quality model predictions against surface observations and build machine learning models to improve the model forecasts. The SMCE has also greatly facilitated collaboration with outside partners, and we plan to use this infrastructure in the future to deliver tailored air quality forecasts directly to stakeholders."

*GMAO Atmospheric Scientist Christoph Keller. NASA photo.*

This animation shows the amount (concentration) of ozone in the atmosphere at the surface of Earth, as represented by the GEOS Composition Forecasting (GEOS-CF) system for the time period July 22–August 10, 2018. High concentrations of ozone are depicted in white while low concentrations are shown in dark blue. Video by NASA's Goddard Space Flight Center.

SMCE - The Future

While NASA and scientists at partner organizations encounter challenges with Big Data and open science, current technologies make moving large volumes of data to the processor environment difficult. As NASA moves more of its data into the cloud, the SMCE managed cloud environment provides scientists with the ability to analyze and visualize data quickly and efficiently, with the data seeming local to the processors.

As the project evolves, the SMCE will support more high-performance computing while striving to provide science as a service in the cloud. The SMCE will continue to accelerate research with a push toward cloud-native science and a cloud-native scientific workforce while working toward consistent environments between on-premises and commercial cloud computing resources.

One of NASA’s strategic goals that the SMCE supports is the Transform to Open Science (TOPS) initiative, which will accelerate the engagement of the scientific community in open science practices while lowering barriers to access and increasing the use of NASA data and previously developed analytics software. The capabilities of the SMCE, along with other NASA strategic initiatives, are needed to fully achieve this transformation.

NASA Center for Climate Simulation

High Performance Computing for Science