[FSDP][1/n] Support LoRA training for FSDP backend.

[GuanxingLu]

Approaches Considered:

1. Merge/Unmerge: Merge LoRA to base -> send merged weights -> unmerge for training.
2. File-based (Current): Save adapter to disk -> rollout engine loads via load_lora_adapter (e.g., AReal).
3. Direct Tensor: Send tensors via memory (e.g., verl).

Current Implementation: Implemented Option 2 using UpdateWeightFromDisk.

Limitation: When rollout engine offloading is enabled, the base model should be re-transmitted in each update (though overhead is manageable).

Status:
✅ Train Qwen3-4B with LoRA on single A100 40G (Full param triggers OOM).
🚧 Reward curve pending.

TODO:

• Implement Options 1 & 3 for comparison.

Update

Current Status:

• Add '--offload-rollout-level' argument, level 1: offload kv cache/cuda graph only, level 2: offload weights + kv cache/cuda graph. When the rollout engine does not release the base model weights, we only need to sync LoRA weights.
• I modified UpdateWeight base class (this should be invasive) to support LoRA update alongside base model weight update.
• Current LoRA weight update logic works for both colocated and disaggregated modes.
• Fixed nits you mentioned before. But I can not remove _lora_loaded because unload_lora_adapter will throw error if LoRA is not loaded.

More TODOs:

• Save/Load logic.
• Efficient layer summon for LoRA weight update.
• Option 3: Send LoRA tensors rather than files.

Discussion Feedback on the trade-offs of these approaches is welcome!

[PopSoda2002]

Great work!

[PopSoda2002]

Has some nit for you :)

[PopSoda2002]

@PopSoda2002 Could you please check this? Thank you!

Current Status:

• Add '--offload-rollout-level' argument, level 1: offload kv cache/cuda graph only, level 2: offload weights + kv cache/cuda graph. When the rollout engine does not release the base model weights, we only need to sync LoRA weights.
• I modified UpdateWeight base class (this should be invasive) to support LoRA update alongside base model weight update.
• Current LoRA weight update logic works for both colocated and disaggregated modes.
• Fixed nits you mentioned before. But I can not remove _lora_loaded because unload_lora_adapter will throw error if LoRA is not loaded.

More TODOs: [] Save/Load logic. [] Efficient layer summon for LoRA weight update. [] Option 3: Send LoRA tensors rather than files.

I moved these to the PR description

[PopSoda2002]

Can you have some test case in wandb compared with no lora? I can help with this part

[GuanxingLu]

Sure, you can run the test. I am also running them in my local setting.

[PopSoda2002]

@GuanxingLu And also, can you please fix the linting? You can use pre-commit

[GuanxingLu]

Sorry, now pre-commit tests are passed.

[Hecate0821]

I am curious how long it take for SGLang to read LoRA weights from disk. Is it possible to pass through NCCL？I am not sure about the file size

[Hecate0821]

And we can not assume that in distributed training, there is a shared file system for every node. So the read from disk approach may not work here

[PopSoda2002]

Can I ask why change this?