AttributeError: no attribute 'base_lrs'

[akaniklaus]

Thank you very much for sharing this impressive work. I am somehow receiving the following error:

    for group, base_lr in zip(self.param_groups, self.base_lrs):
AttributeError: 'AdaBound' object has no attribute 'base_lrs'

[Luolc]

That's very weird...

AdaBound/adabound/adabound.py

Line 43 in 6fa8260

self.base_lrs = list(map(lambda group: group['lr'], self.param_groups))

Once AdaBound is instantiated, the above line would be executed. At least the error shouldn't be has no attribute 'base_lrs'.

Could you provide more details of the environment info? Python version, OS info, and how did you install AdaBound (via pip or direct copy) and which version?

[akaniklaus]

I am getting the error with Python3, PyTorch 1.0 on Ubuntu 18.04 (where I installed Anaconda3) Yes I am not having any error with PyTorch 0.4 so it should be related with the environment

…

On Wed 6. Mar 2019 at 03:29, LoLo ***@***.***> wrote: That's very weird... https://github.com/Luolc/AdaBound/blob/6fa826003f41a57501bde3e2baab1488410fe2da/adabound/adabound.py#L43 Once AdaBound is instantiated, the above line would be executed. At least the error shouldn't be has no attribute 'base_lrs'. Could you provide more details of the environment info? Python version, OS info, and how did you install AdaBound (via pip or direct copy) and which version? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHMEpE1X7ZvPpJ_LQkpy6bCZbs2FFGGbks5vTyf3gaJpZM4beJKp> .

[Luolc]

But it's still very strange because there're only a few minor changes of torch.optim.Adam since the release of version 0.4.1 (I've tested this version and it's fine) on Jul 27, 2018, and they shouldn't have any effects on our code.

I am thinking that there might be some changes in the base class and therefore affect our code. I will have a check with the commits history of PyTorch since 0.4.1 release.

[akaniklaus]

By the way, would you like me to integrate decoupled weight decay into your work? Also, this paper is very relevant to yours: https://openreview.net/forum?id=BJll6o09tm I was actually planning to decay the partial parameter for the same purpose.

…

On Wed 6. Mar 2019 at 08:31, LoLo ***@***.***> wrote: But it's still very strange because there're only a few minor changes <https://github.com/pytorch/pytorch/commits/cf235e0894eb78b741afb98f15da62673ab41dfe/torch/optim/adam.py> of torch.optim.Adam since the release of version 0.4.1 (I've tested this version and it's fine) on Jul 27, 2018, and they shouldn't have any effects on our code. I am thinking that there might be some changes in the base class and therefore affect our code. I will have a check with the commits history of PyTorch since 0.4.1 release. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHMEpJaGxmXUc4A9D67pkNSEmtq3sT3Kks5vT27qgaJpZM4beJKp> .

[Luolc]

Sure. :D
Contributions are welcome as long as you can guarantee the compatibility with current code.

[akaniklaus]

It would be compatible but just let me know if I should directly switch it to 'decoupled weight' decay, which is proven better especially for Adam; or enable the control of it through a boolean? I can also make a seperate function called AdaBoundW for the decoupled weight decay version. Let me know how you would prefer me to proceed on that.

…

On Wed, Mar 6, 2019 at 9:09 AM LoLo ***@***.***> wrote: Sure. :D Contributions are welcome as long as you can guarantee the compatibility with current code. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHMEpFOF04rOJPyRAtL3ufpaVqLp3WnLks5vT3engaJpZM4beJKp> .

[Luolc]

It seems that the decoupled weight decay version for Adam is not integrated into Adam class directly. If so, I think we can just follow them to make a separate function.

[akaniklaus]

Ok will do that.

…

On Wed 6. Mar 2019 at 12:33, LoLo ***@***.***> wrote: It seems that the decoupled weight decay version for Adam is not integrated into Adam class directly. If so, I think we can just follow them to make a separate function. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHMEpPfPUwiryKT-jIOwm1Qg562s9Ms1ks5vT6ehgaJpZM4beJKp> .

[akaniklaus]

Hello @Luolc, FYI, they want to add this to pytorch/contrib if you can make a pull request:
pytorch/pytorch#17560

[Luolc]

Thanks for the reminder! I'll have a look.

I'm now busy with my undergraduate thesis... I'll be available after my graduation, in July.