Question About Which Branch to Fine-tune

Hi, great work! I have a question regarding the fine-tuning process. Since the model adopts a Decomposed Dual-Branch Diffusion architecture—comprising BrushNet and the original UNet—should I fine-tune BrushNet, UNet, or both branches during training?