ENH: Orthogonal LoRA layer initialization (2)

[BenjaminBossan]

Continuation of, and supersedes #2389

Check discussion there for further info.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

I compared the results with orthogonal init vs normal and Gaussian LoRA init, with all other parameters kept equal, using the MetaMathQA method comparison suite. Test accuracy on GSMK8K improved is 47.8% for default, 49.3% for Gaussian, and 48.9% for orthogonal. As expected, memory usage and runtime are practically identical.

I also plotted the train loss:

There seems to be a slight advantage for orthogonal initialization there is well, though for the first half of the run, it lags behind the other methods.