π If you find OpenJudge helpful, please give us a Star! π
OpenJudge is a unified framework designed to drive LLM and Agent application excellence through Holistic Evaluation and Quality Rewards.
π‘ Evaluation and reward signals are the cornerstones of application excellence. Holistic evaluation enables the systematic analysis of shortcomings to drive rapid iteration, while high-quality rewards provide the essential foundation for advanced optimization and fine-tuning.
OpenJudge unifies evaluation metrics and reward signals into a single, standardized Grader interface, offering pre-built graders, flexible customization, and seamless framework integration.
Access 50+ production-ready graders featuring a comprehensive taxonomy, rigorously validated for reliable performance.
|
Focus: Semantic quality, functional correctness, structural compliance Key Graders:
|
Focus: Agent lifecycle, tool calling, memory, plan feasibility, trajectory quality Key Graders:
|
Focus: Image-text coherence, visual generation quality, image helpfulness Key Graders:
|
- π Multi-Scenario Coverage: Extensive support for diverse domains including Agent, text, code, math, and multimodal tasks. π Explore Supported Scenarios
- π Holistic Agent Evaluation: Beyond final outcomes, we assess the entire lifecycleβincluding trajectories, Memory, Reflection, and Tool Use. π Agent Lifecycle Evaluation
- β Quality Assurance: Every grader comes with benchmark datasets and pytest integration for validation. π View Benchmark Datasets
Choose the build method that fits your requirements:
- Customization: Easily extend or modify pre-defined graders to fit your specific needs. π Custom Grader Development Guide
- Data-Driven Rubrics: Have a few examples but no clear rules? Use our tools to automatically generate white-box evaluation criteria (Rubrics) based on your data.π Automatic Rubric Generation Tutorial
- Training Judge Models ( Coming Soonπ): For high-scale and specialized scenarios, we are developing the capability to train dedicated Judge models. Support for SFT, Bradley-Terry models, and Reinforcement Learning workflows is on the way to help you build high-performance, domain-specific graders.
We're actively building seamless connectors for mainstream observability platforms and training frameworks. Stay tuned! β See Integrations
-
2025-12-26 - Released OpenJudge v0.2.0 on PyPI - Major Update! This release expands our core capabilities by adding robust support for diverse evaluation scenarios on top of reward construction. By unifying reward and evaluation signals, OpenJudge v0.2.0 provides a more holistic approach to optimizing application performance and excellence. β migration-guide
-
2025-10-20 - Auto-Rubric: Learning to Extract Generalizable Criteria for Reward Modeling - We released a new paper on learning generalizable reward criteria for robust modeling.
-
2025-10-17 - Taming the Judge: Deconflicting AI Feedback for Stable Reinforcement Learning - We introduced techniques to align judge feedback and improve RL stability.
-
2025-07-09 - Released OpenJudge v0.1.0 on PyPI
pip install py-openjudgeπ‘ More installation methods can be found in the Quickstart Guide.
import asyncio
from openjudge.models import OpenAIChatModel
from openjudge.graders.common.relevance import RelevanceGrader
async def main():
# 1οΈβ£ Create model client
model = OpenAIChatModel(model="qwen3-32b")
# 2οΈβ£ Initialize grader
grader = RelevanceGrader(model=model)
# 3οΈβ£ Prepare data
data = {
"query": "What is machine learning?",
"response": "Machine learning is a subset of AI that enables computers to learn from data.",
}
# 4οΈβ£ Evaluate
result = await grader.aevaluate(**data)
print(f"Score: {result.score}") # Score: 5
print(f"Reason: {result.reason}")
if __name__ == "__main__":
asyncio.run(main())π Complete Quickstart can be found in the Quickstart Guide.
Seamlessly connect OpenJudge with mainstream observability and training platforms, with more integrations on the way:
| Category | Status | Platforms |
|---|---|---|
| Observability | π‘ In Progress | LangSmith, LangFuse, Arize Phoenix |
| Training | π΅ Planned | verl, Trinity-RFT |
π¬ Have a framework you'd like us to prioritize? Open an Issue!
We love your input! We want to make contributing to OpenJudge as easy and transparent as possible.
π¨ Adding New Graders β Have domain-specific evaluation logic? Share it with the community!
π Reporting Bugs β Found a glitch? Help us fix it by opening an issue
π Improving Docs β Clearer explanations or better examples are always welcome
π‘ Proposing Features β Have ideas for new integrations? Let's discuss!
π See full Contributing Guidelines for coding standards and PR process.
OpenJudge was previously distributed as the legacy package
rm-gallery(v0.1.x). Starting from v0.2.0, it is published aspy-openjudgeand the Python import namespace isopenjudge.
OpenJudge v0.2.0 is NOT backward compatible with v0.1.x.
If you are currently using v0.1.x, choose one of the following paths:
- Stay on v0.1.x (legacy): keep using the old package
pip install rm-galleryWe preserved the source code of v0.1.7 (the latest v0.1.x release) in the v0.1.7-legacy branch.
- Migrate to v0.2.0 (recommended): follow the Installation section above, then walk through Quickstart (or the full Quickstart Guide) to update your imports / usage.
If you run into migration issues, please open an issue with your minimal repro and current version.
If you use OpenJudge in your research, please cite:
@software{
title = {OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards},
author = {The OpenJudge Team},
url = {https://github.com/modelscope/OpenJudge},
month = {07},
year = {2025}
}Made with β€οΈ by the OpenJudge Team
![@github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=64&v=4){kind=small}
