diff --git a/README.md b/README.md index 207d5a3..152112b 100644 --- a/README.md +++ b/README.md @@ -9,14 +9,15 @@
- +
## ๐Ÿš€ News -- **[2024.12.18]** We release the **[ArXiv Paper](http://arxiv.org/abs/2412.13147)** of GPassK. ๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰ +- **[2025.1.6]** ๐Ÿ”ฅ **[LiveMathBench](https://huggingface.co/datasets/opencompass/LiveMathBench)** now can be accessed through hugginface, and you can now evaluate your LLMs on it using G-Pass@k in OpenCompass. We have addressed potential errors in LiveMathBench and inconsistencies in the sampling parameters. Please also refer to our updated version of the **[Paper](http://arxiv.org/abs/2412.13147)** for further details. +- **[2024.12.18]** We release the **[ArXiv Paper](http://arxiv.org/abs/2412.13147)** of G-Pass@k. ๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰ ## โ˜€๏ธIntroduction @@ -24,7 +25,7 @@ **G-Pass@k** is a novel evaluation metric that provides a continuous assessment of model performance across multiple sampling attempts, quantifying both the modelโ€™s peak performance potential and its stability. In addition, it comes with **LiveMathBench**, a dynamic benchmark comprising challenging, contemporary mathematical problems designed to minimize data leakage risks during evaluation. In order to track the latest performance and stability of LLMs, we will continue updating the benchmark with new comptition level mathmatical problems and provide the latest results of the models on the benchmark with G-Pass@k. -## ๐ŸŒฒ Definition of GPassK +## ๐ŸŒฒ Definition of G-Pass@k $$ \text{G-Pass@}k = \mathbb{E}_{\text{Questions}} \left[ \frac{{c \choose k}}{{n \choose k}} \right] $$ where $n$ represents the total number of generations per question, and $c$ denotes the number @@ -42,27 +43,95 @@ Intuitively, $\text{mG-Pass@}k$ provides an interpolated estimate of the area un *LiveMathBench-202412 version*
- +
-## ๐Ÿ–‹Use GPassK in OpenCompass +## ๐Ÿ–‹Use G-Pass@k in OpenCompass [OpenCompass](https://github.com/open-compass/opencompass) is a toolkit for evaluating the performance of large language models (LLMs). To use GPassK in OpenCompass, you can follow the steps below: -```python -Coming Soon... + +### 1. Prepare Environment +Follow these steps to ensure your environment is ready: + +```bash +# Clone the main repository +git clone https://github.com/open-compass/GPassK.git +cd GPassK + +# Create and activate a conda environment with specific Python and PyTorch versions +conda create -n livemathbench-eval python=3.10 pytorch torchvision torchaudio pytorch-cuda -c nvidia -c pytorch -y +conda activate livemathbench-eval + +# Install additional required packages +pip install loguru + +# Clone and install OpenCompass for extended functionality +git clone https://github.com/open-compass/opencompass.git opencompass +cd opencompass +pip install -e . +``` + + +### 2. Prepare Dataset +LiveMathBench dataset can be obtained from HuggingFace. First, you should be granted to access the dataset from the following link: [huggingface](https://huggingface.co/datasets/opencompass/LiveMathBench). +Then, refer to [security-tokens](https://huggingface.co/docs/hub/security-tokens) to set up your HF tokens. + + +### 3. Deploy Judge Models +We leverage Qwen2.5-72B-Instruct as the judge model for judging the correctness of generated answers. We recommend to deploy services using deployment tools such as [vllm](https://github.com/vllm-project/vllm) or [lmdeploy](https://github.com/InternLM/lmdeploy) for invocation by different evaluation tasks. + +Below is an example configuration for deploying the judge model using `lmdeploy`: +```bash +lmdeploy serve api_server Qwen/Qwen2.5-72B-Instruct --server-port 8000 \ + --tp 4 \ # at least 4 A100 or equivalent GPUs are required + --cache-max-entry-count 0.9 \ + --log-level INFO +``` +After setting up the judge model, define the URLs in the `eval_urls` within `opencompass_config_templates/*.py`. Adjust other parameters such as `k`๏ผŒ `temperatures`, `llm_infos`, and other params according to your needs. + +> โ—๏ธNote that omitting `eval_urls` will default to an internal rule-based judge, which might only apply to datasets with numerical answers + +### 4. Evaluation + +To begin the evaluation, first generate the necessary configuration files by running the following script: +```bash +python save_opencompass_configs.py --config_template_file {opencompass_config_templates/nono1.py|opencompass_config_templates/o1.py} +``` + +Upon execution, verify the generated configuration files located in `opencompass_configs/: + +``` +. +โ”œโ”€โ”€ deepseek-math-7b-rl_t0-3_p0-8_k50_rp1-0_rs42_l8192@LiveMathBench-v202412-k4_8_16-r3.py +โ”œโ”€โ”€ deepseek-math-7b-rl_t0-5_p0-8_k50_rp1-0_rs42_l8192@LiveMathBench-v202412-k4_8_16-r3.py +โ”œโ”€โ”€ deepseek-math-7b-rl_t0-7_p0-8_k50_rp1-0_rs42_l8192@LiveMathBench-v202412-k4_8_16-r3.py +โ”œโ”€โ”€ deepseek-math-7b-rl_t1-0_p0-8_k50_rp1-0_rs42_l8192@LiveMathBench-v202412-k4_8_16-r3.py +``` + +These files follow a naming convention that reflects the model settings and dataset used: +``` +[MODEL_ABBR]_t[TEMPERATUE]_p[TOP_P]_k[TOP_K]_rp[REPETITION_PENALTY]_l[MAX_OUT_LEN]@[DATASET_ABBR]_k[LIST_OF_K]_r[REPLICATION].py +``` + +With the configurations prepared, initiate the evaluation process with the commands below: + +```bash +cd GPassK +conda activate livemathbench-eval +python opencompass/run.py {path/to/config_file} \ + -w ./opencompass_outputs/ \ + --dump-eval-details \ ``` +Refer to the OpenCompass documentation for additional arguments that may enhance your evaluation experience # Citation and Tech Report -If you use GPassK in your research, please cite the following paper: +If you use G-Pass@k in your research, please cite the following paper: ``` -@misc{liu2024llmscapablestablereasoning, - title={Are Your LLMs Capable of Stable Reasoning?}, - author={Junnan Liu and Hongwei Liu and Linchen Xiao and Ziyi Wang and Kuikun Liu and Songyang Gao and Wenwei Zhang and Songyang Zhang and Kai Chen}, - year={2024}, - eprint={2412.13147}, - archivePrefix={arXiv}, - primaryClass={cs.AI}, - url={https://arxiv.org/abs/2412.13147}, +@article{liu2024your, + title={Are Your LLMs Capable of Stable Reasoning?}, + author={Liu, Junnan and Liu, Hongwei and Xiao, Linchen and Wang, Ziyi and Liu, Kuikun and Gao, Songyang and Zhang, Wenwei and Zhang, Songyang and Chen, Kai}, + journal={arXiv preprint arXiv:2412.13147}, + year={2024} } ``` diff --git a/assets/pass-at-k-v-s-greedy-g-pass-at-k.png b/assets/pass-at-k-v-s-greedy-g-pass-at-k.png new file mode 100644 index 0000000..a0acdc9 Binary files /dev/null and b/assets/pass-at-k-v-s-greedy-g-pass-at-k.png differ diff --git a/assets/performance.png b/assets/performance.png new file mode 100644 index 0000000..06b0914 Binary files /dev/null and b/assets/performance.png differ diff --git a/docs/LiveMathBench-A.csv b/docs/LiveMathBench-A.csv new file mode 100644 index 0000000..38512d0 --- /dev/null +++ b/docs/LiveMathBench-A.csv @@ -0,0 +1,20 @@ +Model,Greedy,G-Pass@16-0.5,G-Pass@16-0.75,G-Pass@16-1.0,mG-Pass@16,link,opensourced,mathLM,o1-like +Llama-3.1-8B-Instruct,24.0,18.2,11.3,4.55,10.4,https://github.com/facebookresearch/llama,TRUE,FALSE,FALSE +Llama-3.1-70B-Instruct,29.8,30.0,22.2,12.5,20.8,https://github.com/facebookresearch/llama,TRUE,FALSE,FALSE +Llama-3.3-70B-Instruct,40.3,36.2,28.9,19.1,27.5,https://github.com/facebookresearch/llama,TRUE,FALSE,FALSE +Qwen2.5-7B-Instruct,37.0,36.5,27.2,16.0,25.8,https://github.com/QwenLM/Qwen,TRUE,FALSE,FALSE +Qwen2.5-32B-Instruct,50.8,48.3,39.5,28.6,38.1,https://github.com/QwenLM/Qwen,TRUE,FALSE,FALSE +Qwen2.5-72B-Instruct,51.7,47.3,39.6,29.0,37.8,https://github.com/QwenLM/Qwen,TRUE,FALSE,FALSE +DeepSeek-V2.5-1210,38.7,38.9,27.9,17.3,26.7,https://github.com/deepseek-ai/DeepSeek-LLM,TRUE,FALSE,FALSE +DeepSeek-V3.0-Chat,55.0,59.5,49.9,35.0,47.9,https://github.com/deepseek-ai/DeepSeek-V3,TRUE,FALSE,FALSE +Mistral-Large-Instruct-2411-123B,41.6,39.4,37.1,32.9,36.4,https://example.com/mistral,TRUE,FALSE,FALSE +Gemini-1.5-Pro-Latest,59.1,55.9,47.3,31.0,44.3,https://example.com/gemini,FALSE,FALSE,FALSE +Claude-3.5-Sonnet,46.7,44.1,36.2,26.6,35.3,https://docs.anthropic.com/claude/docs/models-overview,FALSE,FALSE,FALSE +GPT-4o-2024-11-20,44.8,41.9,32.9,22.2,31.6,https://openai.com/research/gpt-4,FALSE,FALSE,FALSE +DeepSeek-Math-7B-RL,23.5,19.8,14.0,9.7,13.7,https://github.com/deepseek-ai/DeepSeek-LLM,TRUE,TRUE,FALSE +NuminaMath-72B-CoT,40.8,34.0,27.1,14.2,25.0,https://example.com/numinamath,TRUE,TRUE,FALSE +Qwen2.5-Math-7B-Instruct,44.1,44.1,38.3,28.1,36.6,https://github.com/QwenLM/Qwen,TRUE,TRUE,FALSE +Qwen2.5-Math-72B-Instruct,57.6,52.7,45.4,27.9,42.3,https://github.com/QwenLM/Qwen,TRUE,TRUE,FALSE +Skywork-o1-8B,45.4,39.3,31.9,21.7,30.4,https://example.com/skywork,TRUE,FALSE,TRUE +QwQ-32B-Preview,72.7,74.9,65.8,40.1,61.2,https://example.com/qwq,TRUE,FALSE,TRUE +OpenAI o1-mini,74.1,76.3,67.3,48.3,64.8,https://openai.com/research/o1,FALSE,FALSE,TRUE diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 0000000..96ed9ba --- /dev/null +++ b/docs/index.html @@ -0,0 +1,691 @@ + + + + + + + + + + + LiveMathBench Leaderboard + + + + + + + + + + + +
+
+

๐Ÿ† LiveMathBench Leaderboard ๐Ÿ†

+

GPassK: Are Your LLMs Capable of Stable Reasoning?

+
+
+ + description + Paper + + + code + Code + +
+ + + +
+ + +
+
+
+
+ +
+
+
+
+

๐Ÿ“ Notes

+

+

    +
  1. Models labeled with ๐ŸŒ are Closed-source models, while others are Open-sourced.
    2. +
  2. Models labeled with ๐Ÿงฎ are Mathematics-Specialization models.
    3. +
  3. Models labeled with ๐Ÿ’ก are o1-like models with Long-cot.
    4. + +
+

+
+
+
+
+
+
+ + +
+ + + \ No newline at end of file diff --git a/opencompass_config_templates/nono1.py b/opencompass_config_templates/nono1.py new file mode 100644 index 0000000..3b0f494 --- /dev/null +++ b/opencompass_config_templates/nono1.py @@ -0,0 +1,92 @@ +from itertools import product +from mmengine.config import read_base + +with read_base(): + from opencompass.configs.datasets.livemathbench.livemathbench_gen import livemathbench_datasets + +from opencompass.models import TurboMindModelwithChatTemplate + + +k = [4, 8, 16] +replication = 3 +version = '202412' +temperatures = [1.0] + +max_out_len = 8192 +top_p = 0.8 +top_k = 50 +repetition_penalty = 1.0 +random_seed = 42 + +eval_urls = [ + # Put your judge model urls urls here +] + + +llm_infos = [ + # model_name_or_path/tp/batch_size + ('Qwen/Qwen2.5-7B-Instruct', 4, 64), + ('Qwen/Qwen2.5-Math-7B-Instruct', 4, 64), + ('meta-llama/Llama-3.1-8B-Instruct', 8, 64), + ('meta-llama/Llama-3.1-70B-Instruct', 8, 64), + ('meta-llama/Llama-3.3-70B-Instruct', 8, 64), + ('mistralai/Mistral-Large-Instruct-2411', 8, 64), + ('Qwen/Qwen2.5-32B-Instruct', 8, 64), + ('Qwen/Qwen2.5-72B-Instruct', 8, 64), + ('01-ai/Yi-1.5-34B-Chat', 8, 64), + ('deepseek-ai/DeepSeek-V2.5-1210', 8, 16), + ('google/gemma-2-27b-it', 8, 64), + ('deepseek-ai/deepseek-math-7b-rl', 8, 64), + ('internlm/internlm2-math-plus-20b', 8, 64), + ('Qwen/Qwen2.5-Math-72B-Instruct', 8, 64) +] + + +models = [ + dict( + type=TurboMindModelwithChatTemplate, + abbr=llm_info[0].split('/')[-1] + \ + f'_t{temperature}' + f'_p{top_p}' + f'_k{top_k}' + f'_rp{repetition_penalty}' + f'_rs{random_seed}' + f'_l{max_out_len}', + path=llm_info[0], + engine_config=dict(tp=llm_info[1]), + gen_config=dict( + do_sample=False if temperature < 1e-2 else True, + temperature=temperature, + top_p=top_p, + top_k=top_k, + repetition_penalty=repetition_penalty, + random_seed=random_seed + ), + backend='turbomind', + max_out_len=max_out_len, + batch_size=llm_info[2], + run_cfg=dict(num_gpus=llm_info[1]) + ) for llm_info, temperature in product(llm_infos, temperatures) +] + + +livemathbench_dataset = livemathbench_datasets[0] +livemathbench_dataset.update(dict( + k=k, + replication=replication, + dataset_splits=['CNMO', 'CCEE', 'AMC', 'WLPMC'], + dataset_languages=['cn', 'en'], + cot=True, + version=version, + abbr=f'LiveMathBench-v{version}_k{"-".join(map(str, [k] if isinstance(k, int) else k))}_r{replication}' +)) +livemathbench_dataset['eval_cfg']['evaluator'].update(dict( + model_name='Qwen/Qwen2.5-72B-Instruct', + url=eval_urls, + k=k, + replication=replication +)) +livemathbench_dataset['infer_cfg']['inferencer'].update(dict( + max_out_len=max_out_len +)) +datasets = [livemathbench_dataset] \ No newline at end of file diff --git a/opencompass_config_templates/o1.py b/opencompass_config_templates/o1.py new file mode 100644 index 0000000..d73e6e2 --- /dev/null +++ b/opencompass_config_templates/o1.py @@ -0,0 +1,80 @@ +from itertools import product +from mmengine.config import read_base + +with read_base(): + from opencompass.configs.datasets.livemathbench.livemathbench_gen import livemathbench_datasets + +from opencompass.models import TurboMindModelwithChatTemplate + + +k = [4, 8, 16] +replication = 3 +version = '202412' +temperatures = [1.0] + +max_out_len = 32768 +top_p = 0.8 +top_k = 50 +repetition_penalty = 1.0 +random_seed = 42 + +eval_urls = [ + # Put your judge model urls urls here +] + + +llm_infos = [ + # model_name_or_path/tp/batch_size + ('Qwen/QwQ-32B-Preview', 8, 64), + ('Skywork/Skywork-o1-Open-Llama-3.1-8B', 8, 64) +] + + +models = [ + dict( + type=TurboMindModelwithChatTemplate, + abbr=llm_info[0].split('/')[-1] + \ + f'_t{temperature}' + f'_p{top_p}' + f'_k{top_k}' + f'_rp{repetition_penalty}' + f'_rs{random_seed}' + f'_l{max_out_len}', + path=llm_info[0], + engine_config=dict(tp=llm_info[1]), + gen_config=dict( + do_sample=False if temperature < 1e-2 else True, + temperature=temperature, + top_p=top_p, + top_k=top_k, + repetition_penalty=repetition_penalty, + random_seed=random_seed + ), + backend='turbomind', + max_out_len=max_out_len, + batch_size=llm_info[2], + run_cfg=dict(num_gpus=llm_info[1]) + ) for llm_info, temperature in product(llm_infos, temperatures) +] + + +livemathbench_dataset = livemathbench_datasets[0] +livemathbench_dataset.update(dict( + k=k, + replication=replication, + dataset_splits=['CNMO', 'CCEE', 'AMC', 'WLPMC'], + dataset_languages=['cn', 'en'], + cot=True, + version=version, + abbr=f'LiveMathBench-v{version}-k{"_".join(map(str, [k] if isinstance(k, int) else k))}-r{replication}' +)) +livemathbench_dataset['eval_cfg']['evaluator'].update(dict( + model_name='Qwen/Qwen2.5-72B-Instruct', + url=eval_urls, + k=k, + replication=replication +)) +livemathbench_dataset['infer_cfg']['inferencer'].update(dict( + max_out_len=max_out_len +)) +datasets = [livemathbench_dataset] \ No newline at end of file diff --git a/save_opencompass_configs.py b/save_opencompass_configs.py new file mode 100644 index 0000000..3d2de39 --- /dev/null +++ b/save_opencompass_configs.py @@ -0,0 +1,43 @@ +import os +from argparse import ArgumentParser +from typing import List +from itertools import product + +from loguru import logger +from mmengine import Config, mkdir_or_exist + + +def load_and_dumpe_oc_configs(args) -> List[str]: + num_automatic_task = 0 + save_dir = './opencompass_configs' + mkdir_or_exist(save_dir) + + cfg = Config.fromfile(args.config_template_file) + paths = [] + for model_cfg, data_cfg in product(cfg['models'], cfg['datasets']): + save_path = os.path.join(save_dir, + f"{model_cfg['abbr'].replace('.', '-')}" + f"@{data_cfg['abbr']}.py") + automatic_task_cfg = Config(dict(models=[model_cfg])) + automatic_task_cfg.merge_from_dict(dict(datasets=[data_cfg])) + automatic_task_cfg.dump(save_path) + + logger.info(f'|----------> Save opencompass config file to {save_path}') + paths.append(save_path) + num_automatic_task += 1 + + logger.info(f'|----------> Complete saving {num_automatic_task} opencompass config files') + + return paths + + +if __name__ == '__main__': + parser = ArgumentParser() + parser.add_argument('-tc', '--config_template_file', + type=str, + help='the path to opencompass ' + 'config template file') + + args = parser.parse_args() + + load_and_dumpe_oc_configs(args) \ No newline at end of file