Analysis of features in a sliding threshold of observation for numeric
evaluation (STONE) curve
Abstract
We apply idealized scatter-plot distributions to the sliding threshold
of observation for numeric evaluation (STONE) curve, a new model
assessment metric, to examine the relationship between the STONE curve
and the underlying point-spread distribution. The STONE curve is based
on the relative operating characteristic (ROC) curve but is developed to
work with a continuous-valued set of observations, sweeping both the
observed and modeled event identification threshold simultaneously. This
is particularly useful for model predictions of time series data, as is
the case for much of terrestrial weather and space weather. The
identical sweep of both the model and observational thresholds results
in changes to both the modeled and observed event states as the quadrant
boundaries shift. The changes in a data-model pair’s event status result
in nonmonotonic features to appear in the STONE curve when compared to a
ROC curve for the same observational and model data sets. Such features
reveal characteristics in the underlying distributions of the data and
model values. Many idealized datasets were created with known
distributions, connecting certain scatter-plot features to distinct
STONE curve signatures. A comprehensive suite of feature-signature
combinations is presented, including their relationship to several other
metrics. It is shown that nonmonotonic features appear if a local spread
is more than 0.2 of the full domain, or if a local bias is more than
half of the local spread. The example of real-time plasma sheet electron
modeling is used to show the usefulness of this technique, especially in
combination with other metrics.