Training force RMSE increases while energy RMSE decreases -- possible issue with loss prefactors or learning-rate schedule?

[Mem314]

Hi DeePMD developers and community,
I am training a machine-learning interatomic potential for a five-element Si–C–O–H–N system using DeePMD-kit 3.1.2 with the PyTorch backend.
I have encountered a persistent problem: the training force RMSE initially decreases, reaches a minimum, and then starts increasing again during later training steps. At the same time, the energy RMSE generally continues to decrease.
I would appreciate some advice on whether this behavior is expected due to the loss-prefactor schedule, or whether it indicates an issue with my learning rate, training dataset, or model architecture.

1. Training configuration
   My latest experiment uses the se_e2_a descriptor with:

"descriptor": {
"type": "se_e2_a",
"sel": "auto",
"rcut": 6.0,
"rcut_smth": 0.5,
"neuron": [25, 50, 100],
"resnet_dt": true,
"axis_neuron": 16
}

"fitting_net": {
"neuron": [240, 240, 240],
"resnet_dt": true
}

"learning_rate": {
"type": "exp",
"start_lr": 0.0005,
"stop_lr": 3.51e-08,
"decay_steps": 5000,
"scale_by_worker": "none"
}

"loss": {
"type": "ener",
"start_pref_e": 0.1,
"limit_pref_e": 1.0,
"start_pref_f": 1000.0,
"limit_pref_f": 1.0,
"start_pref_v": 0.0,
"limit_pref_v": 0.0
}

The training duration is 800,000 steps.
I have also observed similar force-RMSE behavior in previous experiments using the DPA2 descriptor, with longer training durations and final force prefactors of 1 or 10.

2. Dataset
   My dataset contains crystalline SiC, amorphous SiC, amorphous SiO₂, and various defect-containing Si–C–O–H–N configurations.
   The original combined dataset contained approximately 28,680 configurations distributed across 309 DeePMD systems.
   Some configurations generated during DP-GEN exploration contained very large reference forces, reaching approximately 40 eV/Å.

I initially suspected these high-force configurations were responsible for the increasing force RMSE.
To test this, I created a filtered dataset by removing frames containing atoms with force magnitudes greater than 8 eV/Å.
However, the training force RMSE still increases, even after this filtering.
This makes me suspect that the problem may not be caused primarily by high-force outliers.

3. My suspicion
   I noticed that the force-loss prefactor decreases from 1000 toward 1 as the learning rate decreases, while the energy-loss prefactor increases from 0.1 toward 1.
   Could this cause the optimizer to sacrifice force accuracy in favor of energy accuracy during the later training stages?
   I am also wondering whether my final learning rate of 3.51e-8 is too small, potentially preventing the model from further improving its force predictions.

4. Questions
   I would particularly appreciate advice on the following:

5. Is it normal for training force RMSE to increase while energy RMSE decreases when using DeePMD's default changing loss-prefactor scheme?

6. Could the learning-rate schedule (5e-4 → 3.51e-8) be responsible for this behavior, or is the decreasing force prefactor a more likely explanation?

7. Would you recommend increasing limit_pref_f to 50 or 100, or keeping the energy and force prefactors constant throughout training?

8. For a chemically diverse dataset containing both bulk and defect structures, are there recommended strategies for balancing energy and force fitting?

9. What additional diagnostics would help distinguish a loss-weighting problem from limitations in the training data or model architecture?
   My main objective is to achieve accurate forces for molecular dynamics without significantly compromising energy accuracy.
   I can provide my complete training input.json, learning curves, and relevant log files.
   Thank you very much for your help!

input.json

lcurve.txt

training_log.txt

[sueun-dev]

Part of this is likely the prefactor schedule. In 3.1.2 each loss prefactor follows pref = limit + (start - limit) * lr(t) / start_lr (docs, ener.py). Running the v3.1.2 LearningRateExp with your settings, pref_f is 1000, 303, 92, 9.4 and 1.07 at steps 0, 100k, 200k, 400k and 800k, while pref_e is already 0.92 at 200k. If your force minimum falls between 100k and 400k, the timing matches. Your stop_lr is the same 3.51e-8 as the water example and late updates are tiny, so I doubt it is the driver.

That would explain when energy starts to dominate, not why forces get worse, and your DPA2 runs ending at a force prefactor of 10 showing the same pattern suggest weighting is not the whole story. For the same symptom in #4076, a maintainer pointed (via #2278) to the data section of the precision FAQ: units and force signs, SCF convergence, unstable outliers. dp test on each system would probably show whether a few systems dominate the force error.

If the data checks out, a run with limit_pref_f at 50 to 100 tests the weighting explanation, likely costing some energy accuracy.