Evaluating the potential and challenges of an uncertainty quantification
method for long short-term memory models for soil moisture predictions
Abstract
Recently, recurrent deep networks have shown promise to harness newly
available satellite-sensed data for long-term soil moisture projections.
However, to be useful in forecasting, deep networks must also provide
uncertainty estimates. Here we evaluated Monte Carlo dropout with an
input-dependent data noise term (MCD+N), an efficient uncertainty
estimation framework originally developed in computer vision, for
hydrologic time series predictions. MCD+N simultaneously estimates a
heteroscedastic input-dependent data noise term (a trained error model
attributable to observational noise) and a network weight uncertainty
term (attributable to insufficiently-constrained model parameters).
Although MCD+N has appealing features, many heuristic approximations
were employed during its derivation, and rigorous evaluations and
evidence of its asserted capability to detect dissimilarity were
lacking. To address this, we provided an in-depth evaluation of the
scheme’s potential and limitations. We showed that for reproducing soil
moisture dynamics recorded by the Soil Moisture Active Passive (SMAP)
mission, MCD+N indeed gave a good estimate of predictive error, provided
that we tuned a hyperparameter and used a representative training
dataset. The input-dependent term responded strongly to observational
noise, while the model term clearly acted as a detector for
physiographic dissimilarity from the training data, behaving as
intended. However, when the training and test data were
characteristically different, the input-dependent term could be misled,
undermining its reliability. Additionally, due to the data-driven nature
of the model, the two uncertainty terms are correlated. This approach
has promise, but care is needed to interpret the results.