Understanding and communicating the impact of uncertainty on scientific understanding is a critical unmet need in Earth Science. Challenging as uncertainty determination is for “low level” data, the impact of further processing must be understood, particularly when integrating diverse data types. For example, a fundamental source of diversity is data’s spatiotemporal distribution, for which Point, Grid, and Swath are the most important overarching geographical types. To bring different kinds of data together or bring observational data and model simulations together, observation and simulation values must be “regridded” onto the same grid, i.e. onto a comparable spatiotemporal representation. At the finest level, this requires a detailed understanding of instrumental fields of view and sensitivities. But regridding itself affects uncertainty, especially in situations where significant or irregular interpolation or even extrapolation are required. Furthermore, care must be taken combining diverse data lest the integrated product inherit the worst uncertainty characteristics of each. We have been developing capacities for indexing, regridding, and integrating observation and model simulation that scale to the size and diversity of Earth Science data. In this presentation, we review the fundamental problems associated with combining big, diverse Earth Science data for integrative analysis and how to quantitatively assess and propagate the uncertainties introduced. In particular, uncertainties associated with regridding diverse data for various popular grids and regridding schemes will be assessed.

The variety of missions and observational campaigns in Earth and Space Science has led to a vast number of files containing low-level datasets with native, incompatible arrays. While higher-level datasets re-interpret this low-level data onto common, comparable arrays, this standardization moves scientists farther away from the observations, constraining the analysis and science that can be performed. SpatioTemporal Adaptive Resolution Encoding (STARE) provides a parallelizable, scalable index for co-aligning data with different native spatiotemporal formats efficiently across distributed computing resources. STARE is particularly useful for opening low-level datasets to intercomparison and integrative analysis by providing “array” indexes that carry spatiotemporal semantics, unifying datasets with previously incomparable native array indexing. We are developing STARE as a software library with both C++ and Python APIs and are integrating STARE indexing with existing data transfer tools (OPeNDAP). By organizing data in a hierarchical format and taking advantage of the Hierarchical Data Format’s (HDF) virtualization features, STARE may provide an end user with familiar HDF usability with STARE-enhanced performance and data unification on the back end. Furthermore, STARE’s spatial encoding can be used to index and integrate datasets associated with other planetary bodies, bringing scalability and unification of diverse data for planetary and space science as well.