Configuration features to backtest for epiautogp

[SamuelBrand1]

Hi everyone,

Since epiautogp is getting close to being part of the STF forecast workflow, it seems like a good time to outline what configuration options could lead to greatest out-of-sample performance along with hypotheses of why.

Brief refresher on the underlying AutoGP model

https://probsys.github.io/AutoGP.jl/stable/api.html is a good compact description of the underlying model. At the top level an AutoGP.Model object is an ensemble of Gaussian Processes (GPs) specialised to one dimensional sampling (essentially time -> value) that represents a distribution over possible GP kernel structures (e.g. Sq Exp) and the hyperparameters of that kernel (e.g. length scale) and the observation noise volatility.

A special feature of the package is that as well as common primitive/atomic GP kernels you can have arbitary compositions of kernels via kernel addition, kernel multiplication and "Change point" (CP), which represents one kernel dominating one time segment but this getting smoothly replaced with another kernel.

Terminology

• particles: number of GPs in the ensemble
• SMC: sequential monte carlo. In this context sequentially adding data and possibly resampling/refreshing the ensemble after each new tranche of data is added
• resampling: Drawing a new ensemble of particles by choosing from the current ensemble weighted by their marginal loglik against the current data
• refreshing: Keeping the current particles up to kernel choice but sampling new hyperparams
• MCMC: In this context it means proposing new kernel structures through involutive MCMC and then doing accept/reject on kernel moves
• HMC: In this context its only on hyperparameters conditional on a kernel structure

Main tunable configuration of AutoGP

• Number of particles. Since these represent the uncertainty over the kernel and hyperparamers as an empirical distribution, should we aim for $\mathcal{O}(10^3)$ as per MCMC so that the tail distributions are as expected?
• Number of MCMC steps. If this is higher then there is more opportunity for the particles to sample deeper into the space of possible kernels. Trade off is that this is slower.
• Number of HMC steps. If this is higher then each particle kernels has a high probability of the sampled hyperparams being in a high probability region. Trade off is that this is slower.

Forecasting

Main question for me is whether we should refresh the particles after each forecast sample? The idea behind doing that is that each particle is then forecasting across its posterior distribution (up to kernel conditioning). If we have enough particles is this irrelevant?

Extra tunable modelling choices of NowcastAutoGP

These are about how we approach nowcast samples of data? At the moment, in NowcastAutoGP I have set the default to don't refresh/resample the particles when batch forecasting through nowcast data. Maybe we should though if the nowcasts could potentially be quite extreme.

This is for attention of @dylanhmorris and @damonbayer but if @fsaad has any insider insight I'd love to hear it!

[damonbayer]

outline what configuration options could lead to greatest out-of-sample performance along with hypotheses of why.

@SamuelBrand1 Do you propose making our decision soled based on out-of-sample performance? Or is there some intermediate diagnostics we could evaluate?