Surrogates v2

[pfistfl]

Global

• Clipping to [0,1]

• No tuning over input and output trafos:
  We might want to hardcode it in yahpo_train.

• Checkpoint early stopping

• Dump torch models for post-hoc sanity checks and comparison to ONNX.

• Runtime: How can we ensure monotonicity?
  Idea 1: Train NNs for longer

  Idea 2: Train on diffs (postponed)

  • epoch: Train on diffs and accumulate
  • fraction: Discretize, train on diffs and accumulate.

  Idea 3: Add loss that penalizes f(x) - f(x*) < 0 where x* is a copy of x with increased budget.
  This should be easy to implement, as we can compute v = df(x)i / dxj (target i and feature j) and add a penalty term -lambda * v to the loss.

rbv2

• rbv2_* and iaml_* #65
• Switch to task_id?
• What should the minimum fraction parameter be.
  -> We should evaluate surrogate quality at 0.05 and see if we want to include it.

lcbench

• lcbench #62

iaml

• rbv2_* and iaml_* #65
• Switch to task_id?

fair

• Consider adding fair_* benchmarks #63

nb301

fcnet

• Revisit, perhaps we can find more sensible trafos?

Postponed

• Weight samples by inverse number of obs. per task?
  Either w_i = 1 or w_i = sqrt(1/n_task).

• Revisit noisy surrogates: Revisit noisy #64

[sumny]

@v2_final