Skip to content

Conversation

@lee-groq
Copy link
Contributor

@lee-groq lee-groq commented Dec 4, 2025

Summary

Adds chat-completions/ and responses/ prefixes, providing a universal entrypoint for LLM APIs that support Chat Completions or Responses. This allows openbench to support LLM providers that have not been explictly implemented in openbench. Users add a base URL, API key, and prefix their model ID with chat-completions/ or responses/. This is similar to Inspect AI's openai-api/ prefix.

What are you adding?

  • Bug fix (non-breaking change which fixes an issue)
  • New benchmark/evaluation
  • New model provider
  • CLI enhancement
  • Performance improvement
  • Documentation update
  • API/SDK feature
  • Integration (CI/CD, tools)
  • Export/import functionality
  • Code refactoring
  • Breaking change
  • Other

Changes Made

  • chat_completions.py, which is a generic format provider for Chat Completions API endpoint
  • responses.py, which is a generic format provider for Responses API endpoint
  • Relevant updates to the registry, provider config, and docs

Testing

  • I have run the existing test suite (pytest)
  • I have added tests for my changes
  • I have tested with multiple model providers (if applicable)
  • I have run pre-commit hooks (pre-commit run --all-files)

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (if applicable)
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Related Issues

Closes #

Additional Context


Note

Introduces generic Chat Completions and Responses format providers with new chat-completions/<provider>/<model> and responses/<provider>/<model> prefixes, updates registry/docs/config, and bumps core to 0.5.3.

  • Providers:
    • Add ChatCompletionsFormatAPI (src/openbench/model/_providers/chat_completions.py) and ResponsesFormatAPI (src/openbench/model/_providers/responses.py).
    • Register new prefixes in _registry.py as chat-completions and responses.
    • Extend ProviderType and PROVIDER_CONFIGS with generic entries (env vars CHAT_COMPLETIONS_BASE_URL, RESPONSES_BASE_URL, etc.).
  • Docs:
    • Update unsupported providers guidance to use chat-completions/<provider>/<model-name> or responses/<provider>/<model-name> and adjust example command.
  • Version:
    • Bump openbench-core version to 0.5.3.

Written by Cursor Bugbot for commit 1eb0705. This will update automatically on new commits. Configure here.

@lee-groq lee-groq changed the title feat: add chat-completions and responses provider prefixes feat: add chat-completions/ and responses/ provider prefixes Dec 4, 2025
@lee-groq lee-groq marked this pull request as ready for review December 4, 2025 06:01
@lee-groq lee-groq force-pushed the lee/api-endpoints-provider-prefix branch from a32c217 to 6f90aca Compare December 11, 2025 22:19
@lee-groq
Copy link
Contributor Author

Pardon the messy commits, I had to rebase since the version was bumped from 0.5.2 to 0.5.3 after I originally branched

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants