NEON-FATES-NIWO test in aux_clm fails RUN

[samsrabin]

Brief summary of bug

SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.derecho_intel.clm-FatesFireLightningPopDens--clm-NEON-FATES-NIWO is expected to fail sometimes in SHAREDLIB_BUILD due to Issue #2310: NEON tests sometimes fail because of network issues, but passed that phase for me with ctsm5.3.034. However, it then failed in RUN.

General bug information

CTSM version you are using: ctsm5.3.034.

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: Just that test, as far as I know.

Important output or errors that show the problem

cesm.log:

 ERROR in BalanceCheckMod.F90 at line 986
Image              PC                Routine            Line        Source
cesm.exe           00000000046B5FC4  shr_abort_mod_mp_         106  shr_abort_mod.F90
cesm.exe           00000000046B5F56  shr_abort_mod_mp_          61  shr_abort_mod.F90
cesm.exe           0000000000A4F4A7  abortutils_mp_end          98  abortutils.F90
cesm.exe           000000000116227A  balancecheckmod_m         986  BalanceCheckMod.F90
cesm.exe           0000000000A928A6  clm_driver_mp_clm        1361  clm_driver.F90
cesm.exe           00000000009AB23F  lnd_comp_nuopc_mp         899  lnd_comp_nuopc.F90

lnd.log:

 WARNING:: BalanceCheck, solar radiation balance error (W/m2)
 nstep         =           27
 errsol        =   0.113638626127096
 CTSM is stopping because errsol >   1.000000000000000E-005  W/m2
 fsa           =   0.593138626127096
 fsr           =   0.000000000000000E+000
 forc_solad(1) =   4.250780268302787E-002
 forc_solad(2) =   7.113082344406775E-002
 forc_solai(1) =   0.197242197316972
 forc_solai(2) =   0.168619176555932
 forc_tot      =   0.479500000000000
 coszen_col: -0.167085896483926
 fabd:  0.000000000000000E+000  0.000000000000000E+000
 fabi:  0.000000000000000E+000  0.000000000000000E+000
 albd:  0.000000000000000E+000  0.000000000000000E+000
 albi:  0.000000000000000E+000  0.000000000000000E+000
 ftdd:   1.00000000000000        1.00000000000000
 ftid:   1.00000000000000        1.00000000000000
 ftii:   1.00000000000000        1.00000000000000
 CTSM is stopping
iam = 0: local  patch    index = 2
iam = 0: global patch    index = 2
iam = 0: global column   index = 1
iam = 0: global landunit index = 1
iam = 0: global gridcell index = 1
iam = 0: gridcell longitude    =  254.4167600
iam = 0: gridcell latitude     =   40.0523600
iam = 0: pft      type         = 1
iam = 0: column   type         = 1
iam = 0: landunit type         = 1
 ENDRUN:
 ERROR in BalanceCheckMod.F90 at line 986

Test casedir is at /glade/derecho/scratch/samrabin/tests_0327-213855de/SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.derecho_intel.clm-FatesFireLightningPopDens--clm-NEON-FATES-NIWO.GC.0327-213855de_int/.

[wwieder]

Is this issue related to #3043? Sam, can you reproduce the same test failure issue using ctsm5.3.033?

[samsrabin]

Looks like this passed in the ctsm5.3.033 baseline.

[rgknox]

This will be fixed with: NGEET/fates#1376

[ekluzek]

This same test also fails with izumi_nag and started failing at ctsm5.3.034, with ctsm5.3.033 passing as would be expected.

SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.izumi_nag.clm-FatesFireLightningPopDens--clm-NEON-FATES-NIWO

[ekluzek]

I'm reopening this on the CTSM side, since we haven't updated to the FATES version with the fix in yet. The fix is in sci.1.82.7_api.39.0.0 and ctsm5.3.041 is at sci.1.82.3_api.39.0.0 right now. These tests are still failing in CTSM, until we do bring in a FATES submodule update.