Finetuning (instruction tuning) scripts

[hyunbin70]

Hi, I noticed that you've shared the inference code. Thank you for your effort!

Can you also share the training and fine-tune (instruction tuning) scripts please?
It would be very helpful for my research. thank you!

[BriansIDP]

Hi. Thank you for your interest.

Please keep an eye on https://github.com/bytedance/SALMONN and we will release our training code there shortly.

The only training technique we did not include in the model file is the mixing of single-modal data (e.g. mix audio and video, and the training target is the combination of audio description and video description). Otherwise, all the losses can be found in the model file which can be directly trained in any existing training framework (we specifically used deepspeed).

[XuecWu]

@BriansIDP Hi, thank you for your awesome work! Looking forward to the full release of training code.