Budget check has failed when a planting event

[koxingazhu]

The ED2 running crashed and the error messages showed "Budget check has failed", "Sub-daily budget failed" and "CARBON_FINE: F". I have tried to adjust NL%RK4_TOLERANCE from 0.01 to 0.0001 (because the error messages mentioned RK4 integrator), but it does not work. Anyone have experienced the crash and how to solve it? Thank you!

......

• Simulating: 04/26/2023 00:00:00 UTC
• Simulating: 04/27/2023 00:00:00 UTC
• Simulating: 04/28/2023 00:00:00 UTC
• Simulating: 04/29/2023 00:00:00 UTC
• Simulating: 04/30/2023 00:00:00 UTC
  EVENT 3 of 3

---

---

---

---

      PLANTING 

---

---

---

---

       2   1.00000000    

NEXT EVENT 365 100000000

• Simulating: 05/01/2023 00:00:00 UTC
  |--------------------------------------------------|
  | !!! Sub-daily budget failed !!! |
  |--------------------------------------------------|
  TIME : 2023 05 01 0.
  IPA : 1
  DIST_TYPE : 3
  N_LEAF_RESOLVABLE : 13
  N_WOOD_RESOLVABLE : 12
  NLEV_SFCWATER : 0

AGE : 1.8916664E+01
LAI : 1.8234169E+00
WAI : 4.4496319E-01
VEG_HEIGHT : 1.2103864E+01

Canopy air space properties
..................................................
CAN_DEPTH : 1.2103864E+01
OLD_CAN_PRSS : 9.7650594E+04
CAN_PRSS : 9.7650508E+04
OLD_CAN_ENTHALPY : 3.4314200E+05
CAN_ENTHALPY : 3.4398250E+05
OLD_CAN_TEMP : 2.9756830E+02
CAN_TEMP : 2.9783749E+02
CAN_SHV : 1.4211448E-02
OLD_CAN_SHV : 1.4031631E-02
CAN_CO2 : 3.6485938E+02
OLD_CAN_CO2 : 3.6464517E+02
OLD_CAN_RHOS : 1.1337311E+00
CAN_RHOS : 1.1325827E+00
OLD_CAN_DMOL : 3.8585537E+01
CAN_DMOL : 3.8539421E+01

Summary
..................................................
CO2_FINE : T
CARBON_FINE : F
ENTHALPY_FINE : T
WATER_FINE : T
RADIATION_FINE : T
REL_TOLER_SUBDAY : 1.1920929E-04
REL_TOLER_CARBON : 1.1920929E-03

CO2 Budget
..................................................
TOLERANCE : 2.0595533E+01
RESIDUAL : 1.8707275E-02
INITIAL_STORAGE : 1.7042856E+05
FINAL_STORAGE : 1.7019811E+05
DELTA_STORAGE : -2.3045312E+02
GPP : 2.3392920E+03
LEAF_RESP : 2.5053621E+02
ROOT_RESP : 8.0524769E+00
STORAGE_RESP : 7.0503893E+00
GROWTH_RESP : 3.1476315E+02
HET_RESP : 3.7864053E+02
NEP : 1.3802493E+03
DENSITY_EFFECT : -2.0363263E+02
CANDEPTH_EFFECT : -1.2684250E+02
LOSS2ATM : -1.4802526E+03

Carbon budget
..................................................
TOLERANCE : 8.3607258E-03
RESIDUAL : 1.0737977E-01
INITIAL_STORAGE : 6.7985620E+00
FINAL_STORAGE : 6.9060235E+00
DELTA_STORAGE : 1.0746145E-01
DENSITY_EFFECT : -2.4457704E-06
SEEDRAIN : 6.7869158E-05
CANDEPTH_EFFECT : -1.5234672E-06
LOSS2YIELD : 0.0000000E+00
LOSS2ATM : -1.7778870E-05
COMMITTED : 1.1297570E-03

Enthalpy budget
..................................................
TOLERANCE : 6.8175412E+05
RESIDUAL : 1.5003311E+05
INITIAL_STORAGE : 5.7186570E+09
FINAL_STORAGE : 5.7188127E+09
DELTA_STORAGE : 1.5564800E+05
PRECIPGAIN : 6.2195659E+03
NETRAD : 6.6814227E+04
DENSITY_EFFECT : -4.7752905E+03
PRESSURE_EFFECT : -3.8196540E+00
VEG_HCAP_EFFECT : 2.2997795E+03
CAPACITY_EFFECT : -8.1213708E+02
CANDEPTH_EFFECT : -3.5071450E+03
PHENOLOGY_EFFECT : 8.1002129E+01
LOSS2ATM : 4.2666730E+04
LOSS2DRAINAGE : 1.8034555E+04
LOSS2RUNOFF : 0.0000000E+00

Water budget
..................................................
TOLERANCE : 3.3888975E-01
RESIDUAL : 9.8987088E-02
INITIAL_STORAGE : 2.8426604E+03
FINAL_STORAGE : 2.8427368E+03
DELTA_STORAGE : 7.6416016E-02
PRECIPGAIN : 6.2182546E-03
DENSITY_EFFECT : -1.9668478E-04
CAPACITY_EFFECT : -8.0418878E-04
CANDEPTH_EFFECT : -1.4341282E-04
PHENOLOGY_EFFECT : 5.3232889E-05
LOSS2ATM : 9.4530126E-03
LOSS2DRAINAGE : 1.8245261E-02
LOSS2RUNOFF : 0.0000000E+00

Radiation budget
..................................................
TOLERANCE : 4.5895047E+04
RESIDUAL : 8.0000000E+00
RSHORT_IN : 2.9387626E+07
RSHORT_OUT : 4.9316495E+06
SOIL_RSHORT_NET : 6.7829410E+06
SFCW_RSHORT_NET : 0.0000000E+00
LEAF_RSHORT_NET : 9.9525010E+06
WOOD_RSHORT_NET : 7.7205340E+06
RLONG_IN : 3.3286078E+07
RLONG_OUT : 3.8499556E+07
SOIL_RLONG_NET : 9.3638850E+05
SFCW_RLONG_NET : 0.0000000E+00
LEAF_RLONG_NET : -4.8985150E+06
WOOD_RLONG_NET : -1.2513536E+06

A note on budget check
..................................................
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
STOP fatal_error
Budget failure doesn't necessarily mean a problem in the RK4 or Euler integrators. If you
see NaN in any variable above, and the simulation time is the first day of the month near 00UTC,
then it is very likely that some variable has not been properly initialised in the patch or
cohort dynamics (e.g. new recruit or new patch). The best way to spot the error is to compile the
model with strict debugging options and checks.
Problems that occur in the middle of the day and middle of the month more are more likely to

be true budget violations at sub-daily steps (i.e. thermodynamics, photosynthesis, radiation).

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

---

                 !!! FATAL ERROR !!!                      

---

---> File:        budget_utils.f90
---> Subroutine:  compute_budget
---> Reason:      Budget check has failed, see message above!

---

ED execution halts (see previous error message)...

[koxingazhu]

I am sorry for the poor formatting in the issure, and I do not know how to fix formatting errors. The year is 2023 (2023), doy is 121 (121), event is planting (......), pft=2 (2), density=1.000 (1.000).

[mdietze]

I don't have time to debug, but I'll highlight a bit of text in the error message above:

Budget failure doesn't necessarily mean a problem in the RK4 or Euler integrators. If you
see NaN in any variable above, and the simulation time is the first day of the month near 00UTC,
then it is very likely that some variable has not been properly initialised in the patch or
cohort dynamics (e.g. new recruit or new patch). The best way to spot the error is to compile the
model with strict debugging options and checks.

If the crash is occurring during Planting then it is more likely to be an initialization issue than an integration issue. Even if it's not a completely uninitialized value (NaN), it could be that something is being set-up for the new cohort in a way that is unrealistic or physically implausible that then causes the integrator to go nuts. You'd want to take a closer look at what's triggering the fail to try and figure out what the issue would be.

Also, as an aside, I see that your carbon budget numbers are non-zero for storage respiration -- I strongly advise against ever turning storage respiration on as it's not something that has a physiological basis.

[koxingazhu]

@mdietze Thank you for your suggestions. I have set NL%STORAGE_RESP_SCHEME = 1 (ED-2.2 default, Storage respiration is calculated as in 0......) follow your advice if I understand you correctly and debugged the code. I have found that the model crashed when running on OMP parallel ( !$OMP PARALLEL DO DEFAULT(SHARED) PRIVATE( & in ) within rk4_driver::rk4_timestep function. Dose OpenMP trigger the fail?

I have also noticed the disturbance::plant_patch function when debugging the code and tried to find the initialization issue, but dbh, hgt, agb of new cohort seemed look normal.

When I set Planting PFT is 2/3/4/7/10/11, ED all crashed.
However, ED can run successfully only if I set Planting PFT is 9 - Cold-deciduous early hardwood.

Debug info:
......
(gdb)
336 rk4site%atm_rhv = rehuil8(rk4site%atm_prss,rk4site%atm_tmp,rk4site%atm_shv,.true.)
(gdb)
337 rk4site%atm_rhos = idealdenssh8(rk4site%atm_prss,rk4site%atm_tmp,rk4site%atm_shv)
(gdb)
341 return
(gdb)
342 end subroutine copy_met_2_rk4site
(gdb)
rk4_driver::rk4_timestep (cgrid=<error reading variable: value requires 68056 bytes, which is more than max-value-size>)
at rk4_driver.F90:137
137 !$OMP PARALLEL DO DEFAULT(SHARED) PRIVATE( &

(gdb)
!!! Sub-daily budget failed !!!
--------------------------------------------------
TIME : 2004 07 06 0.
IPA : 1
DIST_TYPE : 3
N_LEAF_RESOLVABLE : 17
N_WOOD_RESOLVABLE : 15
NLEV_SFCWATER : 0

AGE : 8.3333336E-02
LAI : 3.1097009E+00
WAI : 9.5453674E-01
VEG_HEIGHT : 9.9338026E+00
......