[Feature request] SCION optimizer

[sozforex]

Would be nice it to have SCION in this repository for easier comparison.
Here is the official implementation in pytorch https://github.com/LIONS-EPFL/scion

[ClashLuke]

Hey @sozforex, thank you for your interest! I don't currently have the time to add this to HeavyBall. Feel free to submit a PR :)

[ClashLuke]

It's not the full Scion, we now have Scion with norm=Auto:

HeavyBall/heavyball/__init__.py

Lines 203 to 222 in 26f8ba2

	class Scion(C.BaseOpt):
	def __init__(
	self,
	params,
	lr: float = 0.0025,
	betas: tuple[float, float] = (0.9, 0.99),
	eps: float = 1e-8,
	weight_decay: float = 0,
	warmup_steps: int = 0,
	foreach: bool = True,
	storage_dtype: str = "float32",
	mars: bool = False,
	caution: bool = False,
	mars_gamma: float = 0.0025,
	gradient_clipping: C.str_or_fn = C.use_default,
	update_clipping: C.str_or_fn = C.use_default,
	scale: float = 1.0,
	momentum: Optional[float] = None,
	**kwargs,
	):

Let me know if you find any issues or inconsistencies!