[Feature Request] AdaLomo optimizer

[akbar2habibullah]

I use this library a lot, especially to replace Adam with an alternative optimizer. Thank you for the smooth compatibility and pre-tuned settings. Really appreciate your work!

However, I also use another alternative optimizer which this library doesn't support yet, which is AdaLomo. It's not the most performant optimizer but AFAIK is the most memory-efficient optimizer I have ever used. Could you also add the AdaLomo optimizer to the library to support an even more memory-efficient optimizer? Thank you in advance and keep up the great work 🙌

[ClashLuke]

I'm looking into it, thank you for the feature request!