Substorm dynamics in MHD: Statistical validation tests and paths for improvement

Magnetohydrodynamic (MHD) models have been used for nearly four decades to study the dynamics of magnetospheric substorms. However, until recently no demonstration has been made that MHD models can consistently reproduce substorm onset times in a statistical sense. To test whether MHD can reproduce observed substorm onset times, we developed a procedure for identifying substorm onsets that can be applied both to observational data and to MHD output. Our substorm identification procedure aims to improve upon existing methods of substorm identification by using multiple types of observations to corroborate each identified substorm. Using this procedure, we identified over 100 substorms from the period 1-31 January 2005. Using this list of substorm onset times, we show that the MHD model has weak, but statistically significant skill in predicting substorm onset times. We explore paths to improving the ability of the MHD model to predict substorm dynamics by testing different configurations of the MHD model.


METHODOLOGY
We simulated the period 1-31 January 2005. This was chosen due to having a comparatively large number of substorms (between 100 and 300 in previously published lists) and a substantial amount of observational data available. We compiled lists of substorm onset times from the model output and from contemporaneous observational data.

M Mo od de el l d de es sc cr ri ip pt ti io on n
The simulation was performed with the Space Weather Modeling Framework (Toth+ 2005(Toth+ , 2012 with the following components: Block Adaptive Tree Solar-wind Roe-type Upwind Scheme (BATS-R-US) MHD solver (De Zeeuw+ 2000;Powell+ 1999) Ridley Ionosphere Model (Ridley+ 2003(Ridley+ , 2004 Rice Convection Model (Sazykin 2000;Toffoletto+ 2003;Wolf+ 1982) for the inner magnetosphere S Su ub bs st to or rm m i id de en nt ti ifi fic ca at ti io on n All of the available observational datasets have limitations resulting from the placement and availability of observing equipment. At the same time, most of the signatures commonly used to identify substorms can be produced by other magnetospheric processes. As a result, no single dataset is completely reliable for determining when a substorm has occurred. To address this, we developed a procedure for combining lists of substorm onsets from multiple sources.

MHD PREDICTION OF SUBSTORM ONSETS
We computed Heidke skill scores (above) for forecast of substorm onset within a 30 minute window, shown here as a function of frequency bias (the ratio of modeled to observed substorms) for a variety of threshold choices. Values greater than zero represent a skillful forecast; a perfect forecast would have a skill score of one. Error bars denote 95% confidence interval. The skill scores obtained are consistently positive for a wide range of selection parameters. These results show that the MHD model predicts substorms with a skill better than random chance.
Receiver operating characteristic (ROC) curves for forecast of a substorm within a 30-minute window, showing the probability of detection (POD)as a function of probability of false detection (POFD). showing several choices of identification threshold for observed substorms. A perfect forecast would produce a constant POD of 1. The truncation of the curve at POFD~0.2 is a consequence of the procedure used to identify substorms from the data.

SUBSTORM TIMING AND CHARACTERISTICS
Distributions of substorm waiting times from our work and from previous work covering the same time period. Parenthesized numbers in the legend denote the number of substorms in each dataset. We adjusted the tuning parameters of our selection procedure so that it produces a distribution near the median of the previously published datasets.
Superposed epoch analysis of solar wind driving parameters IMF Bz and epsilon, and geomagnetic response parameters AL index and midlatitude positive bay (MPB) index. These plots demonstrate that the model exhibits qualitatively correct behavior with respect to substorm dynamics.

PATHS FOR IMPROVEMENT
The model tends to produce a weaker AL response and greater cross-polar cap potential compared to observations, suggesting that the ionospheric conductivity in the model may be incorrect. We are currently testing the new Magnetosphere -Ionosphere-Thermosphere Conductance Model for the Aurora (MAGNIT) to see how it affects substorm dynamics.
Data assimilation of magnetospheric and ionospheric observations could yield additional improvements. We are developing a new data assimilation system, which is now being tested on SAMI3 (SAMI3 is Another Model of the Ionosphere 3D).
ABSTRACT Magnetohydrodynamic (MHD) models have been used for nearly four decades to study the dynamics of magnetospheric substorms. However, until recently no demonstration has been made that MHD models can consistently reproduce substorm onset times in a statistical sense. To test whether MHD can reproduce observed substorm onset times, we developed a procedure for identifying substorm onsets that can be applied both to observational data and to MHD output. Our substorm identification procedure aims to improve upon existing methods of substorm identification by using multiple types of observations to corroborate each identified substorm. Using this procedure, we identified over 100 substorms from the period 1-31 January 2005. Using this list of substorm onset times, we show that the MHD model has weak, but statistically significant skill in predicting substorm onset times. We explore paths to improving the ability of the MHD model to predict substorm dynamics by testing different configurations of the MHD model.