Any fine-tuning tip?

[tsutsui22]

Hi, I recently try to fine-tune NitroGen model.
It's because NitroGen model looks it does not know what to do, what's enemy, what to shoot, where to go.
Thus, I aimed NitroGen to be accustomed a specific game situation, making it a specialist leveraging its generalist capability.

On fine-tuning, there were several problems.
First, the provided weight here https://huggingface.co/nvidia/NitroGen/blob/main/ng.pt does not have action dim of 21, but 25.
How does action dim of 25 was used? How does mapping between 21 dim of gamepad and action dim set up on the weight?

Since there exists mismatch, I haven't been able to check NitroGen's true capability. Probably, I have failed to fine-tune because of this mismatch. The semantic space that DiT, NitroGen's core, expects as input and output is misaligned with my setting, causing total chaos and can't learn anything meaningfull.

If you have any tips, please teach me, especially how should fine-tuning be conducted.
I'm also curious about learning rate, scheduler setting, what to freeze.
thank you!

[subbon]

Soon I will realese repo for this

[nbopardi]

Soon I will realese repo for this

Hi @subbon -- wanted to reach out to see if you had any updates regarding your repo for fine-tuning NitroGen further.

Thanks!

[JokeryEU]

well i tried using in path of exile 2, lets say after 6 hours didnt pass further than 1 bridge away in the tutorial, avoids enemies, dies a lot, and just button smashing constantly, its more of luck draw if hits the right combo to hit an enemy than actually trying to kill it.