BS PINN Hyper-parametrization

[TUTULEMAN]

Decreased max error by 3 (>50%, currently 2.29) and mean error by 1.4 (>80%, currently 0.44) by applying:

• Learning rate scheduling with exponential decay
• Adaptive loss weighting
• L-BFGS optimizer for fine-tuning
• Increased collocation points

[lamtnguyen989]

Overall, it is good that there is performance gained with tuning. Although there is a few points that can be touched upon which can further help to further improve.

• Firstly, I did see that you fix the epoch=5000 to prevent overfitting of the main training loop. In general, since the objective of the PINN is to minimize the mean-squared error (MSE) of the predictions, it is more effective to save or "checkpoint" the model with the lowest MSE with torch.save() and load it for further training and optimize or even benchmarking with torch.load(). See the KdV, Helmholtz or even the Heat equation note book to see it in action.
• Secondly, the LBGFS-fine tuning is indeed very nice, and does give tangible improvements so if we have more computing power, this is something that we definitely can implement.
• But, based on experience, sometimes it is better to fundamentally change the underlying feedforward Neural Network from the standard model to something else that is better for the job like how the helmholtz model implement the Multiplicative Filter Network to improve performance. But to actually pick the right underlying Network very much comes down to knowing the nature of the problem, the nature of network architecture itself and experience.

Overall, ✨ LGTM ✨ and I'll approve

Few extra resources on PINN architecture:

• Although, this paper mainly focuses on architecture with PINNs that develops shocks, page 11 should give a list of a few PINNs architure worth checking out. (Especially those that deal with the Diffusion equations for Black-Scholes)
• On a more advanced level, you should take a look at the Kolmogorov-Arnold Network which is outlined here and implemented witin the PINN context here.