https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
mgrankin/over9000#4
I strongly believe that AdaBound would be better if it used RAdam instead of Adam.
It could merge with Lookahead too and LAMB.
Then we would have the best of both worlds and a beautiful example of scientific collaboration.