Invalid submissions due to information leakage during TTT

[leloykun]

Don't train on eval tokens your model hasn't scored yet!

---

"Proper" TTT goes like this:

1. For each 1 <= t <= T:
   1.1. Score on eval token t
   1.2. Adapt weights based on eval token <= t

---

What y'all are doing is something like this:

1. For each 1 <= t <= T:
   1.1. Adapt weights based on eval token <= t
2. For each 1 <= t <= T:
   2.1. Score on eval token t

But this is equivalent to appending the eval tokens to the training tokens and switching training strategies before eval! Also see: #152 (comment)

---

Potentially invalid submissions:

PR	comment	status
#136	TTT on half of the batch; eval on full batch	[ ] open
#152	TTT on all eval tokens before evaluation	[x] closed
#254	TTT on multiple parts of eval sequence for multiple epochs	[ ] open
#264	TTT before eval	[ ] open
#338	TTT on multiple parts of eval sequence for multiple epochs	[ ] open
#398	TTT on all eval tokens before evaluation	[ ] Open
#421	TTT before evals	[ ] open
#417	TTT for multiple epochs before evals	[ ] open
#442	TTT before evals	[ ] open

cc @0hq

---

Please feel free to correct me if I'm wrong.

[leloykun]

I've also noticed that people are doing TTT on multiple parts of the evaluation sequence in parallel (via DDP rank), justifying it by only assigning each "doc" to one rank.

I'm not sure if this is comparable to the default eval strategy where we just feed all eval tokens to the model... wdyt @0hq ?

[himanalot]

this is clearly happening bc people are using agents to monitor/use others' prs to improve their strategies. it's had an exponentially strong trickledown effect ever since a few of them made submissions that appear good at first glance with this method of basically cheating. it's a bit odd to watch.