Request for Training/Evaluation Configs Used in Qwen3 Experiments

[WeiYang2023]

Great work on the paper “Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasets”! We’re particularly interested in performing our experiments on the Qwen3-series models. Could you kindly share the corresponding training and evaluation configuration files? This would help us align our experiments with those reported in the paper.

Thanks in advance!

[GX-XinGao]

Thanks for your interest in the paper and for the kind words!

Regarding the Qwen3-series experiments, we’d like to clarify where the training and evaluation settings come from and how to reproduce them:

1. Paper & ODA report (background reference)
   The experimental settings used in “Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasets” are described in Section 2.4 and Section 3.3 of the paper.
   The same unified training and evaluation protocol is also summarized in Appendix A of our ODA main report (“OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value”). These sections explain the controlled setup and design choices behind the experiments.

2. GitHub configs (directly reproducible)
   For practical use and reproducibility, we provide the exact configuration files in the GitHub repository:

• Qwen3 training config:
  https://github.com/OpenDataArena/OpenDataArena-Tool/blob/main/model_train/train_config/qwen3_config.yaml
  This YAML file corresponds to the training setup used in the paper and can be directly reused or adapted for Qwen3-series models.
• Qwen3 evaluation setup:
  All benchmarks and their corresponding OpenCompass Benchmark Files are listed in
  https://github.com/OpenDataArena/OpenDataArena-Tool/blob/main/model_eval/README.md
  To reproduce the full evaluation, you can directly run:
  https://github.com/OpenDataArena/OpenDataArena-Tool/blob/main/model_eval/eval_script/test_qwen3_all_benchmarks.sh

Importantly, the settings described in the paper, the ODA main report, and the GitHub configs are fully consistent with each other. We intentionally keep a single, unified configuration across documentation and code to ensure strict fairness and objective comparison across datasets and models.

If you have further questions when running the configs or adapting them to a specific Qwen3 variant, feel free to follow up—we’re happy to help.

Thanks again for your interest in ODA and the data-loop work!

[WeiYang2023]

Thanks for your interest in the paper and for the kind words!

Regarding the Qwen3-series experiments, we’d like to clarify where the training and evaluation settings come from and how to reproduce them:

1. Paper & ODA report (background reference)
   The experimental settings used in “Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasets” are described in Section 2.4 and Section 3.3 of the paper.
   The same unified training and evaluation protocol is also summarized in Appendix A of our ODA main report (“OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value”). These sections explain the controlled setup and design choices behind the experiments.
2. GitHub configs (directly reproducible)
   For practical use and reproducibility, we provide the exact configuration files in the GitHub repository:

• Qwen3 training config:
  https://github.com/OpenDataArena/OpenDataArena-Tool/blob/main/model_train/train_config/qwen3_config.yaml
  This YAML file corresponds to the training setup used in the paper and can be directly reused or adapted for Qwen3-series models.
• Qwen3 evaluation setup:
  All benchmarks and their corresponding OpenCompass Benchmark Files are listed in
  https://github.com/OpenDataArena/OpenDataArena-Tool/blob/main/model_eval/README.md
  To reproduce the full evaluation, you can directly run:
  https://github.com/OpenDataArena/OpenDataArena-Tool/blob/main/model_eval/eval_script/test_qwen3_all_benchmarks.sh

Importantly, the settings described in the paper, the ODA main report, and the GitHub configs are fully consistent with each other. We intentionally keep a single, unified configuration across documentation and code to ensure strict fairness and objective comparison across datasets and models.

If you have further questions when running the configs or adapting them to a specific Qwen3 variant, feel free to follow up—we’re happy to help.

Thanks again for your interest in ODA and the data-loop work!

Thank you for your detailed reply! I have a couple of questions regarding the reproduction of the Qwen3 series:

1. Do we need to update the versions of vllm, OpenCompass, or any other dependencies in the project?
2. It seems that some configuration files—such as aime2024_repeat8_cver_gen and aime2025_repeat8_cver_gen—are missing or cannot be found in the current repository.
   Thanks in advance for your help!

[orangeadegit]

Thanks for the follow-up!

1. Dependencies: You do not need to update or install external versions.

• The repository already includes modified source code for both vLLM and OpenCompass.
• These included versions are based on vLLM 0.8.5 and OpenCompass 0.4.2.
• We explicitly modified the vLLM code to address infinite loop/repetitive output issues that can occur during the model's "thinking" process. Please simply use the versions provided in the repo.

2. Missing Configs: We have added the missing configuration files. You can now find them at the following paths:

• model_eval/opencompass/opencompass/configs/datasets/aime2024/aime2024_repeat8_cver_gen.py
• model_eval/opencompass/opencompass/configs/datasets/aime2025/aime2025_repeat8_cver_gen.py

Let me know if you run into any other issues!

[WeiYang2023]

Thanks for the follow-up!

1. Dependencies: You do not need to update or install external versions.

• The repository already includes modified source code for both vLLM and OpenCompass.
• These included versions are based on vLLM 0.8.5 and OpenCompass 0.4.2.
• We explicitly modified the vLLM code to address infinite loop/repetitive output issues that can occur during the model's "thinking" process. Please simply use the versions provided in the repo.

2. Missing Configs: We have added the missing configuration files. You can now find them at the following paths:

• model_eval/opencompass/opencompass/configs/datasets/aime2024/aime2024_repeat8_cver_gen.py
• model_eval/opencompass/opencompass/configs/datasets/aime2025/aime2025_repeat8_cver_gen.py

Let me know if you run into any other issues!

Thanks for the detailed reply! I’ll give it a try.