feat: add CI/CD workflow to validate tagging of MCQEval tasks #330
Conversation
|
✅ Benchmark documentation has been automatically updated. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| sys.path.insert(0, str(Path(__file__).parent.parent / "src")) | ||
|
|
||
| from openbench.config import get_all_benchmarks, BENCHMARKS | ||
| from openbench.utils.mcq import get_mcq_benchmarks |
There was a problem hiding this comment.
Bug: Missing function causes validation script to fail
The script imports get_mcq_benchmarks from openbench.utils.mcq, but this function doesn't exist in that module or anywhere in the codebase. The script will fail immediately with an ImportError when executed, breaking the entire CI/CD workflow. The documentation mentions using is_mcq_task() to detect MCQ benchmarks, but neither get_mcq_benchmarks nor is_mcq_task are implemented.
|
|
||
| # Replace in content | ||
| if old_tags_str in content: | ||
| content = content.replace(old_tags_str, new_tags_str, 1) |
There was a problem hiding this comment.
Bug: String replacement modifies wrong benchmark with duplicate tags
The script uses content.replace(old_tags_str, new_tags_str, 1) to modify tags, which replaces only the first occurrence of a tag list in the file. When multiple benchmarks share identical tag lists (like the 14 MMMLU language variants all having ["mcq", "knowledge", "multilingual", "mmmlu"]), the script will modify the first matching benchmark instead of the intended one. This causes incorrect tag modifications where the wrong benchmarks get updated while the intended benchmarks remain unchanged.
Additional Locations (1)
|
This PR is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. |
Summary
Adds CI/CD validation and enforcement of "mcq" tags for all tasks implemented with the MCQEval task factory.
Note: Decided to implement as a CI/CD feature instead of a pre-commit hook to avoid bogging down dev process.
What are you adding?
Changes Made
BenchmarkMetadataobjects inconfig.py.validate-mcq-tags.ymlfor post-commit workflow, trigger on changes to MCQ-related files andconfig.pyTesting
pytest)pre-commit run --all-files)Checklist
Note
Adds a CI workflow and script to detect and auto-fix MCQ tags, standardizes tags to
mcqacross config/docs, and updates the Arabic Exams group (40 tasks)..github/workflows/validate-mcq-tags.ymlto validate and auto-fixmcqtags, commit changes tosrc/openbench/config.py, and comment results on PRs.scripts/validate_mcq_tags.pyto detect MCQ benchmarks (viaget_mcq_benchmarks/is_mcq_task()), report missing/incorrectmcqtags, auto-editconfig.py, and revalidate; supports alpha-only/no-alpha flags."multiple-choice"with"mcq"tags acrosssrc/openbench/config.pyanddocs/snippets/benchmarks.data.mdx.arabic_exams_math_high_school, set count/description to 40 tasks.Written by Cursor Bugbot for commit 5a22517. This will update automatically on new commits. Configure here.