Through the NESDIS Cloud Archive Project (NCAP), NESDIS is building on previous cloud pilot efforts and making significant progress on delivering an end-to-end cloud archive prototype in late 2021 and operational cloud archive capability in 2022. Critical to the overall design is a clear mapping between the features being designed and the functional requirements driven by National Archives and Records Administration (NARA), the Open Archival Information System Reference Model (OAIS-RM), and the NESDIS vision for a consolidated cloud archive workflow. The mapping between these drivers and the features being developed using Agile methodologies will be reviewed, and a technical summary of NESDIS’ plans to implement data archiving using commercial cloud services will be provided.

In the two years since the Fall 2019 meeting, NOAA/NESDIS completed two significant projects whose goal was to prototype an enterprise data management system in the cloud. The two pilot efforts, run by the NESDIS Cloud Integrated Project Team, were completed in March of 2020 and March of 2021. A significant portion of those pilot efforts has been operationalized into what is known as the NESDIS Common Cloud Framework (NCCF). The NCCF is now up and running and capable of securely ingesting data and generating operational products. Another effort, the NESDIS Cloud Archive Project, picked up the unfinished work of the NESDIS cloud pilots and is focused on delivering an end-to-end cloud archive prototype in late 2021, with the goal of operationalizing that cloud archive capability into the NCCF in 2022. The current status of these efforts will be presented, including a technical focus on how NESDIS plans to archive data using commercial cloud services.

Since its formation in 2015, the National Centers for Environmental Information (NCEI) has used disparate, legacy systems spread across several IT networks of the National Environmental Satellite, Data, and Information Service (NESDIS) to fulfill its data-stewardship functions. As part of its modernization and consolidation of these functions, NCEI implemented Common Ingest as the functional component that ingests approximately 200 data streams every month into its enterprise archival information system. In parallel, NESDIS completed the Secure Ingest Gateway Project (SIGP), a pilot project to establish standard-enterprise secure methods for NESDIS and the rest of the National Oceanic and Atmospheric Administration (NOAA) to receive data in a cloud environment from their external partners. SIGP is now transitioning to operations as the Operational Secure Ingest Service (OSIS), which will be the on-ramp to NCEI’s “Common Ingest” functionality when it too moves to the cloud. In addition, this ingest function will populate and use a cloud-based metadata catalog, which will be the beating heart of the NESDIS and NCEI information systems in the cloud environment. The vision is to scale their ingest of environmental data to keep pace with its ever increasing volume, veracity, variety, and velocity. In this presentation to the ocean data community, the authors describe NESDIS and NCEI’s challenges and successes with the implementation of the ingest function of their archival information system in a cloud environment.