The training performance of basic MLLM RLVR configuration is slow

[canghongjian]

I ran your basic MLLM RLVR configuration examples/qwen2.5-vl-7B-rlvr/rlvr_megatron.yaml, but the training performance was almost stand, swanlab url:
https://swanlab.cn/@canghongjian/web_public/runs/n5yko5po0dgktwbl9zs0d/chart

while the test performance also has no change after training:

	RefCOCO_test	CountBenchQA	RefCOCO_g_test	RefCOCO_plus_test	MathVista_MINI	MathVerse_MINI	CountQA_test
Qwen2_5_VL_7B_Instruct	0.8992	0.8789	0.8624	0.8113	0.6850	0.4508	0.2075
vlm_roll_rlvr_ckpt_20	0.8976	0.8727	0.8610	0.8089	0.6950	0.4114	0.2055
vlm_roll_rlvr_ckpt_100	0.8970	0.8645	0.8606	0.8147	0.6940	0.4363	0.2016

[HuangJoJo]

@guoshengCS plz take a look at this issue

[guoshengCS]

It seems that the detection score is exceptionally low and even compared with the pasted test performance. Intuitively, I may suspect that the exceptionally low detection score might be caused by unmatched bbox ground-truth and detection reward implementation. The detection reward implementation requires the ground-truth bbox to be scaled to [0,1000]. You can check this when using custom datasets. @canghongjian

[canghongjian]

@guoshengCS Thanks for your reply! I used the following configuration. Did you mean that I used the wrong dataset?

actor_train:
  model_args:
    flash_attn: fa2
    attn_implementation: fa2
    disable_gradient_checkpointing: false
    dtype: bf16
    model_type: ~
  training_args:
    learning_rate: 1.0e-6
    weight_decay: 1.0e-2
    per_device_train_batch_size: 2
    gradient_accumulation_steps: 256
    warmup_steps: 0
    num_train_epochs: 50
  data_args:
    # use One-RL-to-See-Them-All/Orsta-Data-47k as train dataset
    # download from https://huggingface.co/datasets/One-RL-to-See-Them-All/Orsta-Data-47k
    file_name:
      - /publicdata/huggingface.co/datasets/One-RL-to-See-Them-All/Orsta-Data-47k/main/train/train_detection_v3det_4000.parquet
      - /publicdata/huggingface.co/datasets/One-RL-to-See-Them-All/Orsta-Data-47k/main/train/train_math_mmmath_3539.parquet
    domain_interleave_probs:
      math: 0.5
      cv_detection: 0.5
    dataset_dir: ./
    messages: prompt
    preprocessing_num_workers: 32
  strategy_args:
    strategy_name: megatron_train
    strategy_config:
      sequence_parallel: true
      tensor_model_parallel_size: 4
      context_parallel_size: 2
      expert_model_parallel_size: 1
      pipeline_model_parallel_size: 1
      overlap_grad_reduce: true
      use_distributed_optimizer: true
      bf16: true
  device_mapping: list(range(0,32))
  infer_batch_size: 8

[guoshengCS]

@guoshengCS Thanks for your reply! I used the following configuration. Did you mean that I used the wrong dataset?

actor_train:
  model_args:
    flash_attn: fa2
    attn_implementation: fa2
    disable_gradient_checkpointing: false
    dtype: bf16
    model_type: ~
  training_args:
    learning_rate: 1.0e-6
    weight_decay: 1.0e-2
    per_device_train_batch_size: 2
    gradient_accumulation_steps: 256
    warmup_steps: 0
    num_train_epochs: 50
  data_args:
    # use One-RL-to-See-Them-All/Orsta-Data-47k as train dataset
    # download from https://huggingface.co/datasets/One-RL-to-See-Them-All/Orsta-Data-47k
    file_name:
      - /publicdata/huggingface.co/datasets/One-RL-to-See-Them-All/Orsta-Data-47k/main/train/train_detection_v3det_4000.parquet
      - /publicdata/huggingface.co/datasets/One-RL-to-See-Them-All/Orsta-Data-47k/main/train/train_math_mmmath_3539.parquet
    domain_interleave_probs:
      math: 0.5
      cv_detection: 0.5
    dataset_dir: ./
    messages: prompt
    preprocessing_num_workers: 32
  strategy_args:
    strategy_name: megatron_train
    strategy_config:
      sequence_parallel: true
      tensor_model_parallel_size: 4
      context_parallel_size: 2
      expert_model_parallel_size: 1
      pipeline_model_parallel_size: 1
      overlap_grad_reduce: true
      use_distributed_optimizer: true
      bf16: true
  device_mapping: list(range(0,32))
  infer_batch_size: 8

The datasets used here is okay. We will validate the configuration again to check