-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Is your feature request related to a problem? Please describe.
When running a large evaluation, the same question/answer pair might be evaluated multiple times. Each time, an expensive and relatively slow API call is made to the AI judge.
Describe the solution you'd like
Implement a caching mechanism for the _acall_llm_judge method in the RAGEvaluator.
- It could be a simple in-memory cache for a single run.
- It could be a more advanced on-disk cache (e.g., using
shelveorsqlite) that persists between runs.
Describe alternatives you've considered
The alternative is to continue making redundant API calls, which is inefficient.
Additional context
This would be a significant performance and cost-saving improvement for users evaluating large test sets.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels