feat: Add hosted evals options. #303
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Introduces a platform-hosted evaluation flow alongside local runs, with CLI ergonomics and async polling/log streaming.
--hostedmode inprime eval runand deprecatedprime env eval; requiresowner/nameslug and resolves environment ID from hub--poll-interval,--no-stream-logs,--timeout-minutes,--allow-sandbox-access,--allow-instances-access,--custom-secrets,--eval-nameutils/hosted_eval.pyprovidingHostedEvalConfig,HostedEvalResult,run_hosted_evaluation(create via/hosted-evaluations, poll/evaluations/{id}, stream logs, fetch final stats) andprint_hosted_resultenv.run_evalto parse JSON args (--env-args,--custom-secrets), create hosted config, run viaasyncio, print results, and treat non-COMPLETEDas failure; retains existing local eval path and installation behaviorWritten by Cursor Bugbot for commit 1d4747c. This will update automatically on new commits. Configure here.