Theemulationoffutureclimatestatespresentsparticularchallengesformachinelearningandotherstatisticalapproaches.Chieflyamongthoseisthelimitedamountoftrainingdatathatistypicallyavailable;currentMLapproachesarenotpreparedtolearnsuchcomplexscenariosinsmalldataregimesunderacovariateshift.Aspointedout,thecomplexESMsthataretrustedtomodelthefutureclimateareextremelycomputationallyexpensivetorunandtheobservationalrecordcannotinformusaboutunseenfuturescenarios.ByharnessingalargeselectionofsimulationsperformedaspartofCMIP6,ClimateBenchattemptstoalleviatethisdifficulty,butneverthelessonlyaround500trainingpoints(years)representrealisticclimatestates,manyofwhicharenotindependent(asshowninFig.A1).Thispresentsachallengefordeeplearningapproacheswhichtypicallyrequiretensofthousandsoftrainingsamplestoavoidover-fitting.Theinclusionoflongeridealisedsimulationsdoesprovideopportunitiesforpre-traininghowever,particularlythe500-yearlongpiControlsimulationswhichcouldbeusedwithcontrastivelearningtoreducethetrainingsamplesrequiredforneuralnetworkarchitectures.ThepiControlsimulationcouldalsobeusedtoinformemulatorsmoreexplicitlyabouttheinternalvariabilityofclimate(asproducedbyNorESM2).Thesignal,particularlyfortheprecipitationtargetvariables,canbesmallcomparedtothisvariabilityandthisproveschallengingforsomeemulatorstoreproduce.Anexplicitmodeloftheinternalvariability(Castruccioetal.,2019)couldhelptoalleviatethis.Anotherchallengeinapplyingstatisticallearningapproachestothisdatasetistherelativelyhighdimensionalinputsandoutputs(96x144).MostapproachestoemulatingtheregionaltemperatureresponsetoaCO2forcinghavebeencarriedoutat,atmost,dozensoflocations,butaccountingforthespatialcorrelationsissomethingwhichCNNscanexcelatandhaverecentlybeenshowntoproduceaccurateemulationsoftemperatureacrosssimilardimensionality(Beuschetal.,2020).Suchapproachestypicallyassumearegularspacing,however,andneglectthereducingareaofeachgrid-celltowardsthepoles.Whilemoretraditionalapproachesofdimensionalityreductioncanalsobeused,suchas(weighted)empiricalorthogonalfunctions(EOFs),thesemaynotbeappropriateforthenon-linearprecipitationfieldswhichmightrequirekernel-basedapproximations(e.g.,Buesoetal.,2019).Forpracticalpurposes,anestimateoftheuncertaintyinanypredictionwouldbeextremelyvaluable.Thisuncertaintyshouldencompassthatduetotheinternalvariabilityandtheemulatorapproximation(andideallythatoftheunderlyingphysicalmodel).IntheMLcommunity,theseareknownastheepistemicandthemodeluncertainties,andarebeingstudiedintensively(Kendalletal.,2017).Quantifyingthesetwouncertaintieswouldallowincreasedtrust(aconceptexploredinthenextsection)inthepredictionaswellasquantitativecomparisontootherpredictions.Weencouragetheestimationofuncertaintywhereverpossible,usingtheprovidedCRPSmetrictoevaluatesuchprobabilisticprojections.Theabilitytosamplefromsuchdistributionswouldalsopermitthegenerationofso-called‘superensembles’whichcanprovideverylargeensemblesofmultiplemodelsundergivenscenarios(Beuschetal.,2020).

Emulatortrustworthiness

Forclimatemodelemulatorstobeusefulforpolicydecisionstheymustbetrustedbytheirusers.Thetrustworthinessofanymodelisasubjectiveconceptthatbroadlyrepresentsone’sbeliefthatthemodelfaithfullyrepresentssomeunderlying‘truth’.Modelverificationattemptstoobjectivelyassertthisview(indeedthewordderivesfromtheLatin,verus,meaningtrue)butisformallyimpossibleforanopensystemliketheEarth(seee.g.,Oreskesetal.,1994).Whileweathermodelscanberegularlyvalidatedagainstobservations,intheclimatesciencesweofteninsteadresorttonecessarilyincompletemodelevaluationandrelyonunderlyingphysicalprinciplestoprovidereassurancesofbroadervalidity.TheClimateBenchemulatorsside-stepthisissuebyaimingonlytoaccuratelyreproduceanexistingphysicalmodelwhichisassumedtoalreadybewellevaluated,andthereforeattaintrustworthinessthroughproxy.Itwouldneverthelessbereassuringiftheemulatorscouldbedemonstratedtorespectsomeofthesamephysicalconstraints.Inthisspirit,Figure6showstherelativechangeinglobalmeanprecipitationasafunctionofglobalmeantemperaturechange(thehydrologicalsensitivity)ofthebaselineemulatorsandNorESM2.WhilelocallyprecipitationcanchangeinaccordancewiththeClausius-Clapeyronrelationship(6-7%/K),energyconservationrequiresthattheglobalchangesinprecipitationarebalancedbyradiativecoolingandlimitedto2-3%/K(AllenandIngram,2002;PendergrassandHartmann,2013;JeevanjeeandRomps,2018;Daganetal.,2019).WhiletheRFemulatorunderestimatesthehydrologicalsensitivityofNorESM,itisclearthattheemulatorslearnthephysicalrelationshipfromtheunderlyingmodel.Sincetheemulatorsweretrainedontheprecipitationandtemperaturethisistobeexpectedtosomedegree,butthisdemonstratestheprinciplethatemulatorstrainedcorrectlycanretainthephysicallawsoftheunderlyingmodels.Futureeffortstointroducetheseinvariancesdirectlyhavethepotentialtosignificantlyeasethetrainingandimprovetheinferenceofclimatemodelemulators(Beucleretal.,2021),ultimatelyimprovingtheirtrustworthiness.