AdamW performing Adam without weight decay?

[tom-jod]

The weight decay parameter is set by default to 0, and I can't see where it is assigned a value other than this. When I printed out the group["weight_decay"] in chain it returns 0.0 for every step. Please can you let me know if I am missing something here. Thank you

[ClashLuke]

That sounds like a severe issue, thank you for raising this. That might explain some of #58.

[ClashLuke]

I've tested the weight decay code, and it seems correct. Am I understanding you correctly that the confusion is that the weight decay value is to 0 by default and that the value still is 0 inside the algorithm? That's intentional.

The difference between AdamW and Adam isn't the addition of weight decay, it's that the weight decay is decoupled. Standard Adam would instead treat weight decay as an L2 penalty, which means the weight decay modifies the momentum values.

However, your obsevation that the default differ is on point and this should be documented better. It's definitely a sharp edge

[tom-jod]

Ah I see, thank you very much for clearing that up! So, if I just run --opt AdamW how is the amount of weight decay chosen?(this is an important hyper parameter, no?)

[ClashLuke]

In the HeavyBall benchmark? Currently, weight decay is always set to 0. You'd have to double-check whether some tasks get unfair advantages through it (such as rastrignin) and likely shift their target value. But good point, this should definitely be tuned as well!