We present a book entitled “Machine Learning, Statistics, and Data Mining for Heliophysics,” an online and open source book available at helioml.org. This book includes a collection of interactive Jupyter notebooks, written in the programming language Python, that walks the reader through the process of applying machine learning, statistics, and data mining techniques on various kinds of solar and space physics data sets to reproduce published results. We consider this book to be a living document with frequent updates. Please contact us if you’d like to submit a chapter!

A sudden surge of data has created new challenges in water management, spanning quality control, assimilation, and analysis. Few approaches are available to integrate growing volumes of data into interpretable results. Process-based hydrologic models have not been designed to consume large amounts of data. Alternatively, new machine learning tools can automate data analysis and forecasting, but their lack of interpretability and reliance on very large data sets limits the discovery of insights and may impact trust. To that end, we present a new approach, which seeks to strike a middle ground between process-, and data-based modeling. The contribution of this work is an automated and scalable methodology that discovers differential equations and latent state estimations within hydrologic systems using only rainfall and runoff measurements. We show how this enables automated tools to learn interpretable models of 6 to 18 parameters solely from measurements. We apply this approach to nearly 400 stream gaging sites across the US, showing how complex catchment dynamics can be reconstructed solely from rainfall and runoff measurements. We also show how the approach discovers surrogate models that can replicate the dynamics of a much more complex process-based model, but at a fraction of the computational complexity. We discuss how the resulting representation of watershed dynamics provides insight and computational efficiency to enable automated predictions across large sensor networks.

As phytoplankton form the base of the marine food web, understanding the controls on their abundance is fundamental to understanding marine ecology and how it might be altered by global climate change. While many Earth System Models (ESMs) predict phytoplankton biomass, it is unclear whether they properly capture the mechanistic relationships that control this quantity in the real ocean. In this paper, we used Random Forest (RF) analysis to analyze the output of ESMs and observational datasets. We gathered information from 13 ESMs and two observational datasets. The target variable was phytoplankton carbon and the predictors included environmental parameters known to influence phytoplankton, such as nutrients, light, mixed layer depth, salinity, temperature, and upwelling. We examined three questions: (1) What fractions of variability in ESMs and observations can be linked to the large-scale environmental variables simulated by ESMs? (2) What are the dominant predictors and relationships affecting phytoplankton biomass? (3) How well do ESMs simulate phytoplankton carbon and do they simulate the relationships we see in observations? We show that about 88% to 96% of the variability in observational datasets and greater than 98% in the ESMs was accounted for by variables known to influence phytoplankton biomass from large-scale environmental variables. The dominant predictors in the observational datasets were dissolved iron and shortwave radiation. The dominant predictors in the ESMs were dissolved iron, shortwave radiation, and mixed layer depth. While relationships in most of the ESMs matched the general trends seen in the observations, significant quantitative differences were seen. While the assumption made by ESMs that large-scale environmental conditions control phytoplankton biomass appears to hold in the real world, much work remains to be done to ensure that ESMs properly represent these controls.

In an active volcanic arc, magmatically sourced fluids are channeled through the brittle crust by structural features. This interaction is observed in the Andean volcanic mountain belt, where volcanoes, geothermal springs and the locations of major mineral deposits coincide with NNE-striking, convergent margin-parallel faults and margin-oblique, NW/SE-striking Andean Transverse Faults (ATF). The Tinguiririca and Planchón-Peteroa volcanoes in the Andean Southern Volcanic Zone (SVZ) demonstrate this relationship, as both volcanic complexes and their spatially associated thermal springs show strike alignment to the outcropping NNE oriented El Fierro Thrust Fault System. This study aims to constrain the 3D architecture of this fault system and its interaction with volcanically sourced hydrothermal fluids from a combined magnetotelluric (MT) and seismicity survey. The 3D conductivity model and seismic hypocenter locations show correlations between strong conductivity contrasts and seismic clusters in the top 10km of the crust. This includes a distinct WNW-striking seismogenic feature which has characteristics of the ATF domains. As the surveyed region is characterized by high heat flow regimes, volcanic activity and hydrothermal systems related to the volcanic arc, the conductivity contrast suggests that magmatically derived fluids meet an impenetrable barrier, most likely the sealed core of the fault. The resulting increase in hydrostatic fluid pressure facilitates seismic activity on this WNW oriented structure. These results provides the first observation of the mechanism behind the reactivation and seismogenesis of ATF. The study also uncovers the role of the ATF the compartmentalization of magmatic-derived fluids that accumulate to form hydrothermal reservoirs in the SVZ.

The objective of this work was to understand the suitability and favorability of saline reservoirs for storing hydrogen and ensuring the maximum amount of hydrogen would be withdrawn. We used the Sacramento basin in California to demonstrate the feasibility. We carried out several numerical simulation studies to understand key factors affecting the storage and withdrawal of hydrogen. We combined the results from the numerical modeling to develop a screening and ranking set of criteria for hydrogen storage in saline reservoirs. We then used the screening and ranking set of criteria to rank the formations in the Sacramento basin. We studied five formations in the Sacramento Basin. The numerical simulation study showed that to optimize storage and withdrawal of hydrogen, steeply dipping reservoirs up to 15 degrees, reservoirs with low pressures, reservoirs with good porosity (above 20%), and reservoirs with high permeabilities were most favorable for underground hydrogen storage. This work applies a novel and comprehensive site screening and selection criteria for underground hydrogen storage in saline reservoirs. It builds on a similar work done for carbon dioxide storage and hydrogen storage in depleted gas fields but it considers additional objective of needing to withdraw the stored fluid. The case study in Sacramento Basin can be applied to any other basin.

Liquefaction of saturated erodible bed sediments is often indicated as one of the main drivers for the bulking of geophysical mass flows and their extensive mobility, but this is especially difficult to capture in numerical models. In this work, we employ a two-phase 3D MPM–CFD model to simulate a granular column collapsing over a water-saturated erodible bed. The two-phase model explicitly simulates the solid and fluid phases and their interactions, thereby allowing the natural emergence of pore pressures, liquefaction, erosion, and flow runout within the simulations. Our results suggest that the initial dilatancy and permeability of the bed play crucial roles in governing the dynamics. Contractive beds with lower permeability exhibit larger pore pressure buildup and erosion, resulting in longer runout compared to dilative beds, consistent with field observations. These findings suggest the importance of considering the hydro-mechanical properties of erodible bed materials for predictive modelling of landslide hazards.

An ensemble-based method for wave data assimilation is implemented using significant wave height observations from the globally distributed network of Sofar Spotter buoys and satellite altimeters. The Local Ensemble Transform Kalman Filter (LETKF) method generates skillful analysis fields resulting in reduced forecast errors out to 2.5 days when used as initial conditions in a cycled wave data assimilation system. The LETKF method provides more physically realistic model state updates that better reflect the underlying sea state dynamics and uncertainty compared to methods such as optimal interpolation. Skill assessment far from any included observations and inspection of specific storm events highlight the advantages of LETKF over an optimal interpolation method for data assimilation. This advancement has immediate value in improving predictions of the sea state and, more broadly, enabling future coupled data assimilation and utilization of global surface observations across domains (atmosphere-wave-ocean).

This study presents a workflow to predict the upscaled absolute permeability of the rock core direct from CT images whose resolution are not sufficient to allow direct pore-scale permeability computation. This workflow exploits the deep learning technique with the data of raw CT images of rocks and their corresponding permeability value obtained by performing flow simulation on high resolution CT images. The permeability map of a much larger region in the rock core is predicted by the trained neural network. Finally, the upscaled permeability of the entire rock core is calculated by the Darcy flow solver, and the results showed a good agreement with the experiment data. This proposed deep-learning based upscaling method allows estimating the permeability of large-scale core samples while preserving the effects of fine-scale pore structure variations due to the local heterogeneity.

Compound extremes exhibit greater adverse impacts than their univariate counterparts. Studies have reported changes in frequency and the spatial extent of extremes in India; however, investigation of compound extremes is in the infancy state. This study investigates the historical variation of compound dry and hot extremes (CDHE) and compound wet and cold extremes (CWCE) during the Indian summer monsoon period from 1951-2019 using monthly data. Results are analyzed for 10 identified homogeneous regions for India. Our results unraveled that CDHE (CWCE) frequency has increased (decreased) by 1-3 events per decade for the recent period (1977-2019) relative to the base period (1951-1976). Overall, there is increasing (decreasing) pattern of CDHE (CWCE) is high across North-central India, Western India, North-eastern India and South-eastern coastlines. Our findings help in identification of the parts of the country were affected by frequent and widespread CDHE during the recent period, which is alarming. More detailed assessments are required to disentangle the complex physical process of compound extremes to improve risk management options.

A direct comparison of hydrometeor types (HMT) from state-of-the-art hydrometeor classification schemes (HMC) with modelled hydrometeors (ICOL-LAM, operational weather predictions model of the German Weather Service) is challenging, e.g. due to different HMT definitions and numbers and difficulties to identify dominant types in mixtures of hydrometeors. A comparison of published HMCs even revealed significant differences between the membership functions used for the same hydrometeor types (Figure 1), emphasizing again the high uncertainty in scattering simulations for ice hydrometeors because of their complex geometries, dielectric properties, and largely unknown size and orientation distributions. The HMCs were applied to perturbed polarimetric variables observed by the X-band Radar in Bonn (BoXPol) to test their robustness against measurement errors and show that especially in the regions with solid precipitation misclassification in hydrometeor typing occurs often. Thus, a dual strategy to evaluate the hydrometeor type representation in ICON-LAM is presented: i) Classification after clustering of the data is assumed to reduce the sensitivity of the decision to the uncertainty of scattering simulations. First an agglomerative hierarchical clustering of the radar pixels based on their similarity in multi-dimensional polarimetric signatures is applied, and afterwards for each identified cluster a comparison of the distributions of polarimetric moments with scattering simulations or membership functions for different HMT is performed. ii) A direct comparison of multivariate simulated and observed distributions of polarimetric moments. These comparisons will be performed for different heights and/or space-time subsets, and for clusters with similar HMT in the model and the observations as identified with the advanced radar-based hydrometeor classification scheme. Results for a set of case studies observed with the polarimetric X-band radar composite in Bonn, Germany, will be presented.

Solar flare, as a typical large temporal-spatial turbulent magnetic reconnection (LTSTMR, the ratio of observed current sheets thickness to electron characteristic length, electron Larmor radius for low-β and electron inertial length for high-β, is on the order of 10E10–10E11; the ratio of observed evolution time to electron gyroperiod is on the order of 10E7–10E9) explosion in the solar atmosphere activities, involving sudden bursts of particle acceleration that from sudden release of magnetic energy in a few minutes to a few tens of minutes. The X rays and gamma rays are believed to result from the interactions of the high energy electrons energized and nuclear interaction of the high energy protons and other heavier ions, respectively. While many particle acceleration models consider turbulence acceleration as an effective way of generating energetic electrons, the precise turbulence roles during acceleration and heating of electrons still remain unclear. Here we show from 3D relativistic hybrid particle-in-cell and lattice Boltzmann method (RHPIC-LBM) simulation that interaction of helical magnetic structure that leads to efficient energization of electrons. By following the trajectories of the most energetic electrons, we found the strong Langmuir turbulence acceleration (LTA) through wave-wave, wave-particle interaction in the diffusion region of the flare, which can accelerate electrons effectively. and discuss the turbulence acceleration by strong Langmuir wave. The simulation of LTA is not only similar to the shock wave acceleration, but more efficient than that of the shock wave acceleration. The energy spectrum of hot electrons undergoing LTA can be studied the X ray and gamma ray production in flare. We anticipate our results to be a key point for understanding the relationship between particle acceleration mechanism and explosive energetic electrons observed in the solar flares during MHD Alfven turbulence translate into Kinetic Alfven turbulence progress.

Quantifying mixing rates in the Arctic Ocean is critical to our ability to predict heat flux, freshwater distribution, and circulation. However, turbulence measurements in the Arctic are sparse, and cannot characterize the high spatiotemporal variability typical of ocean mixing. Using year-round temperature and salinity data from Ice-Tethered Profiler (ITP) instruments between 2004 and 2018, we apply a finescale parameterization to obtain pan-Arctic estimates of turbulent dissipation and mixing rates at unprecedented space-time resolution. Building on previous work that used ITP data to identify double-diffusive staircases and analyze the associated convective mixing, we apply the finescale parameterization only where these step-like thermohaline structures are not present and mixing is expected to be internal wave-dominated. We find that the inferred wave-driven dissipation and mixing rates are generally low, but highly variable in both space and time, displaying significant regional differences between the shelves and central basins, as well as a small seasonal cycle. We detect no statistically significant interannual trend in mixing rate estimates over the period examined, with the exception of a small increase in the Canada Basin immediately below the mixed layer. The joint consideration of turbulent dissipation rates and stratification imply varied Arctic Ocean mixing regimes, which are most often not appropriately characterized as isotropic turbulence. Where justified, we infer turbulent heat fluxes out of the Atlantic Water layer that are mostly small, but also exhibit a distinct regional dependence.

Studies of interior air exposures to various human and non-human components has largely been restricted to industrial exposures for the purpose of regulation. In contrast, little attention has been paid to exposure at the residential scale, where people spend much of their day and may be exposed to particulate sources ranging from known toxins, such as lead, arsenic, and asbestos, to human-produced chemicals of yet unknown toxicity, such as flame retardants. To capitalize on experience with citizen science initiatives as they pertain to environmental health, researchers formed an international network called 360 Dust Analysis, which provides guidance on citizen science and interior dust collection, as well as research tools to examine dust through analysis in regional labs. We present initial results from the July 2018 launch of this program in the USA, called DustSafe USA and operated under approved human subjects protocols by Indiana University (http://www.urbanhealth.iupui.edu/). We launched via multiple media strategies, including an extended television news segment, an article in several Indiana newspapers, appearances in several statewide radio shows, and via a widely distributed press release. As of this abstract submission, well over 300 queries were received, and after only two weeks of the launch the lab has received nearly 100 dust samples. Participants are largely from central Indiana where most of the media play occurred, but samples have also come from all over the country. We will present geochemical and compositional results from the dust analysis, but perhaps more importantly we will discuss how citizens were engaged, how the funding model for such efforts might be developed, and the general approach to research translation and citizen science.

A document by Navid Hosseini. Click on the document to view its contents.

Weather and climate forecast predictability relies on Land-Atmosphere (L-A) interactions occurring at different time scales. However, evaluation of L-A coupling parameterizations in current land surface models (LSMs) is challenging since the physical processes are complex, and large-scale observations are scarce and uncommon. Recent advancements in satellite observations, in this light, provide a unique opportunity to evaluate the models’ performances at large spatial scales. Using 5-year soil moisture memory (SMM) from Soil Moisture Active and Passive (SMAP) observations, we evaluate L-A coupling performances in 4 prevailing LSMs with both coupled and offline simulations. Multi-model mean comparison at the global scale shows that current LSMs tend to overestimate SMM that is controlled by water-limited processes and vice versa. Large model spreads in SMM are also observed between individual models. The SMM biases are highly dependent on models’ parameterizations, while showing minor relevance to the models’ soil layer depths or the models’ online/offline simulating schemes. Further analyses of two important terrestrial water cycle-related variables indicate current LSMs may underestimate soil moisture that is directly available for evapotranspiration and global flood risks. Finally, a comparison of two soil moisture thresholds indicates that the soil parameters employed in LSMs play an essential role in producing the model’s biases. The satellite estimation of ET at the water-limited stage and soil hydraulic parameters provides readily available information to constrain LSMs, which are essentially important to improve the models’ L-A coupling simulations, as well as other land surface processes such as terrestrial hydrological cycles.

We perform a suite of laboratory friction experiments on saw-cut Westerly Granite surfaces and probe frictional state evolution in response to step changes in normal stress. The experiments are conducted with the objective of illuminating the origin of friction memory effects and the fundamental processes that yield friction rate and state dependence. In contrast to previous works, we measure directly the fault slip rate and account for changes in slip rate caused by normal stress perturbations. Further, we complement mechanical data acquisition by continuously probing the faults with ultrasonic pulses. We conduct the experiments at room temperature and humidity conditions in a servo controlled biaxial testing apparatus in the double direct shear configuration. The normal stress perturbations are carried out during steady shearing over a range of shear velocities, from 0.02 - 100 μm/s. We report observations of a transient shear stress and friction evolution with step increases and decreases in normal stress. Specifically, we show that shear stress evolves in a two-stage fashion – first linear-elastically, then inelastically in response to the normal stress step. We find that the excursions in slip rate resulting from the changes in normal stress must be accounted for in order to accurately predict fault strength evolution. The effects of induced changes in fault slip rate are also apparent in elastic wave properties. Ultrasonic wave amplitudes increase instantly in response to normal stress steps and then gradually decrease to a new steady state value, in part due to changes in fault slip rate. This decrease is strongly related to accelerated creep at the fault interface. We also demonstrate that steady state amplitudes are a reliable proxy for real contact area (RCA) at the fault interface. Previous descriptions of frictional state evolution during normal stress perturbations have not adequately accounted for large slip velocity excursions. Here, we do so by using the measured ultrasonic amplitudes as a proxy for frictional state during transient shear stress evolution. Our work improves understanding of induced seismicity and triggered earthquakes with particular focus on simulating static triggering and stress transfer phenomena using rate-and-state frictional formulations in earthquake rupture models.

Better understanding of the hydrological cycle of the Lower Mekong River Basin (LMRB) and addressing the value-added information of using remote sensing data on the spatial variability of soil moisture over the Mekong Basin is the objective of this work. In this work, we present the development and assessment of the LMRB (drainage area of 495,000 km2) Soil and Water Assessment Tool (SWAT). The coupled model framework presented is part of SERVIR, a joint capacity building venture between NASA and the U.S. Agency for International Development, providing state-of-the-art, satellite-based earth monitoring, imaging and mapping data, geospatial information, predictive models, and science applications to improve environmental decision-making among multiple developing nations. The developed LMRB SWAT model enables the integration of satellite-based daily gridded precipitation, air temperature, digital elevation model, soil texture, and land cover and land use data to drive SWAT model simulations over the Lower Mekong River Basin. The LMRB SWAT model driven by remote sensing climate data was calibrated and verified with observed runoff data at the watershed outlet as well as at multiple sites along the main river course. Another LMRB SWAT model set driven by in-situ climate observations was also calibrated and verified to streamflow data. Simulated soil moisture estimates from the two models were then examined and compared to a downscaled Soil Moisture Active Passive Sensor (SMAP) 36 km radiometer products. Results from this work present a framework for improving SWAT performance by utilizing a downscaled SMAP soil moisture products used for model calibration and validation.

Moderate earthquakes (Mw > 5) with moment tensors (MTs) dominated by a vertical compensated-linear-vector-dipole (vertical-CLVD) component are often generated by dip slip along a curved ring-fault system at active volcanoes. However, relating their MTs to ring-fault parameters has been proved difficult. The objective of this study is to find a robust way of estimating some ring-fault parameters based on their MT solutions obtained from long-period seismic records. We first model the MTs of idealized ring-faulting and show that MT components representing the vertical-CLVD and vertical strike-slip mechanisms are resolvable by the deviatoric MT inversion using long-period seismic waves, whereas a component representing the vertical dip-slip mechanism is indeterminate owing to a shallow source depth. We then propose a new method for estimating the arc angle and orientation of ring-faulting using the two resolvable MT components. For validation, we study a vertical-CLVD earthquake that occurred during the 2005 volcanic activity at the Sierra Negra caldera, Galápagos Islands. The resolvable MT components are stably determined with long-period seismic waves, and our estimation of the ring-fault parameters is consistent with the ring-fault geometry identified by previous geodetic studies and field surveys. We also estimate ring-fault parameters of two earthquakes that took place during the 2018 activity at the caldera, revealing significant differences between the two earthquakes in terms of slip direction and location. These results show the usefulness of our method for estimating ring-fault parameters, enabling us to examine the kinematics and structures below active volcanoes with ring faults that are distributed globally.