Challenges and Successes of Implementing NOAA/NESDIS/NCEI’s Ingest of
Archival Data in a Cloud Environment
Abstract
Since its formation in 2015, the National Centers for Environmental
Information (NCEI) has used disparate, legacy systems spread across
several IT networks of the National Environmental Satellite, Data, and
Information Service (NESDIS) to fulfill its data-stewardship functions.
As part of its modernization and consolidation of these functions, NCEI
implemented Common Ingest as the functional component that ingests
approximately 200 data streams every month into its enterprise archival
information system. In parallel, NESDIS completed the Secure Ingest
Gateway Project (SIGP), a pilot project to establish standard-enterprise
secure methods for NESDIS and the rest of the National Oceanic and
Atmospheric Administration (NOAA) to receive data in a cloud environment
from their external partners. SIGP is now transitioning to operations as
the Operational Secure Ingest Service (OSIS), which will be the on-ramp
to NCEI’s “Common Ingest” functionality when it too moves to the
cloud. In addition, this ingest function will populate and use a
cloud-based metadata catalog, which will be the beating heart of the
NESDIS and NCEI information systems in the cloud environment. The vision
is to scale their ingest of environmental data to keep pace with its
ever increasing volume, veracity, variety, and velocity. In this
presentation to the ocean data community, the authors describe NESDIS
and NCEI’s challenges and successes with the implementation of the
ingest function of their archival information system in a cloud
environment.