loading page

Creating and evaluating uncertainty estimates with neural networks for environmental-science applications
  • +2
  • Katherine Haynes,
  • Ryan Lagerquist,
  • Marie McGraw,
  • Kate Musgrave,
  • Imme Ebert-Uphoff
Katherine Haynes
Cooperative Institute for Research in the Atmosphere
Author Profile
Ryan Lagerquist
Cooperative Institute for Research in the Atmosphere

Corresponding Author:[email protected]

Author Profile
Marie McGraw
Cooperative Institute for Research in the Atmosphere
Author Profile
Kate Musgrave
Cooperative Institute for Research in the Atmosphere
Author Profile
Imme Ebert-Uphoff
Cooperative Institute for Research in the Atmosphere
Author Profile

Abstract

Neural networks (NN) have become an important tool for prediction tasks -- both regression and classification -- in environmental science.  Since many environmental-science problems involve life-or-death decisions and policy-making, it is crucial to provide not only predictions but also an estimate of the uncertainty in the predictions.  Until recently, very few tools were available to provide uncertainty quantification (UQ) for NN predictions.  However, in recent years the computer-science field has developed numerous UQ approaches, and several research groups are exploring how to apply these approaches in environmental science.  We provide an accessible introduction to six of these UQ approaches, then focus on tools for the next step, namely to answer the question: Once we obtain an uncertainty estimate (using any approach), how do we know whether it is good or bad?  To answer this question, we highlight four evaluation graphics and eight evaluation scores that are well suited for evaluating and comparing uncertainty estimates (NN-based or otherwise) for environmental-science applications.  We demonstrate the UQ approaches and UQ-evaluation methods for two real-world problems: (1) estimating vertical profiles of atmospheric dewpoint (a regression task) and (2) predicting convection over Taiwan based on Himawari-8 satellite imagery (a classification task).  We also provide Jupyter notebooks with Python code for implementing the UQ approaches and UQ-evaluation methods discussed herein.  This article provides the environmental-science community with the knowledge and tools to start incorporating the large number of emerging UQ methods into their research.
06 Dec 2022Submitted to ESS Open Archive
07 Dec 2022Published in ESS Open Archive