suggestion: kl divergence only #8
Description
I was thinking that the kl divergence that rlhf and rlvr both use to encourage readability of the results. could be the cause and is actually making the training process into an annealing process where, the output must stay the same, but the internal signals that are passed through the system are actually more. practically then the random rewards would be a method of grokking search. Assuming that the robust circuits are the ones that are most likely to be correct.
And as such I suggest you make a training process where you force the model to try to keep the output the same, while adding noise to the gradients or simply using dropout?
Someone else suggested that grpo would act as a denoiser in this case.