[FEATURE] Validate endpoints in the cli

[burtenshaw]

Summary

This PR adds a validation feature to the cli which works like this:

openenv validate https://openenv-wordle.hf.space

output

{
  "target": "https://openenv-wordle.hf.space",
  "validation_type": "running_environment",
  "standard_version": "1.0.0",
  "standard_profile": "openenv-http/1.x",
  "mode": "simulation",
  "passed": false,
  "criteria": [
    {
      "id": "openapi_version_available",
      "description": "GET /openapi.json returns OpenAPI info.version",
      "passed": true,
      "required": true,
      "expected": {
        "status_code": 200,
        "info.version": "string"
      },
      "actual": {
        "status_code": 200,
        "info.version": "1.0.0"
      }
    },
    {
      "id": "health_endpoint",
      "description": "GET /health returns healthy status",
      "passed": true,
      "required": true,
      "expected": {
        "status_code": 200,
        "status": "healthy"
      },
      "actual": {
        "status_code": 200,
        "status": "healthy"
      }
    },
    {
      "id": "metadata_endpoint",
      "description": "GET /metadata returns name and description",
      "passed": true,
      "required": true,
      "expected": {
        "status_code": 200,
        "fields": [
          "name",
          "description"
        ]
      },
      "actual": {
        "status_code": 200,
        "name": "TextArenaEnvironment",
        "description": "TextArenaEnvironment environment"
      }
    },
    {
      "id": "schema_endpoint",
      "description": "GET /schema returns action, observation, and state schemas",
      "passed": true,
      "required": true,
      "expected": {
        "status_code": 200,
        "fields": [
          "action",
          "observation",
          "state"
        ]
      },
      "actual": {
        "status_code": 200,
        "has_action": true,
        "has_observation": true,
        "has_state": true
      }
    },
    {
      "id": "mcp_endpoint",
      "description": "POST /mcp is reachable and returns JSON-RPC payload",
      "passed": false,
      "required": true,
      "expected": {
        "status_code": 200,
        "jsonrpc": "2.0"
      },
      "actual": {
        "status_code": 404,
        "jsonrpc": null
      }
    },
    {
      "id": "mode_endpoint_consistency",
      "description": "OpenAPI endpoint set matches OpenEnv mode contract",
      "passed": true,
      "required": true,
      "expected": {
        "/reset": true,
        "/step": true,
        "/state": true
      },
      "actual": {
        "/reset": true,
        "/step": true,
        "/state": true
      }
    }
  ]
}

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• New environment
• Refactoring

Alignment Checklist

Before submitting, verify:

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated
• I have run /pre-submit-pr (or bash .claude/hooks/lint.sh and tests) and addressed all issues

RFC Status

• Not required (bug fix, docs, minor refactoring)
• RFC exists: #___
• RFC needed (will create before merge)

Test Plan

Claude Code Review

[greptile-apps]

Greptile Summary

This PR adds runtime validation to the openenv validate CLI command, enabling validation of live OpenEnv servers via URL. The implementation is well-architected and aligns with OpenEnv principles.

Major changes:

• New validate_running_environment() function validates HTTP endpoints (/openapi.json, /health, /metadata, /schema, /mcp) and returns structured JSON reports
• Validates simulation vs production mode by checking endpoint consistency: simulation mode requires /reset, /step, /state; production mode must not expose these infrastructure endpoints
• CLI accepts URLs via positional argument or --url flag, maintains backward compatibility with local path validation
• Adds --json flag for structured output from local validation
• Comprehensive test coverage with mocked HTTP responses

Alignment with principles:

• Correctly enforces dual API boundary by validating that simulation controls (/reset, /step, /state) are only present in simulation mode
• Uses existing requests dependency (already in pyproject.toml)
• Well-tested with both success and failure scenarios
• Backward compatible with existing local validation workflow

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• Score reflects well-tested implementation with comprehensive coverage, no security issues, proper error handling, and alignment with system invariants. The validation logic correctly enforces the dual API boundary by checking simulation vs production mode endpoint consistency.
• No files require special attention

Important Files Changed

Filename	Overview
src/openenv/cli/_validation.py	Adds validate_running_environment() to check live OpenEnv servers against runtime API standards, validates simulation vs production mode endpoint consistency, and adds JSON report generation
src/openenv/cli/commands/validate.py	Extends openenv validate command to accept URLs (via argument or --url flag), adds --json output option, and maintains backward compatibility with local path validation
tests/test_cli/test_validate.py	Comprehensive test coverage for runtime validation with mocked HTTP responses, local validation, JSON output, and error cases

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[openenv validate] --> B{Target type?}
    B -->|URL / --url flag| C[validate_running_environment]
    B -->|Local path| D[validate_multi_mode_deployment]
    
    C --> C1[Check /openapi.json]
    C1 --> C2[Check /health]
    C2 --> C3[Check /metadata]
    C3 --> C4[Check /schema]
    C4 --> C5[Check /mcp]
    C5 --> C6{Has /reset endpoint?}
    C6 -->|Yes| C7[Mode: simulation<br/>Expect /step, /state]
    C6 -->|No| C8[Mode: production<br/>Expect no /step, /state]
    C7 --> C9[Generate JSON report]
    C8 --> C9
    
    D --> D1[Check pyproject.toml]
    D1 --> D2[Check uv.lock]
    D2 --> D3[Check server/app.py]
    D3 --> D4[Check dependencies]
    D4 --> D5{--json flag?}
    D5 -->|Yes| D6[Output JSON report]
    D5 -->|No| D7[Output text report]
    
    C9 --> E[Exit with status]
    D6 --> E
    D7 --> E

Loading

_{Last reviewed commit: 4709978}

[wukaixingxp]

was able to run this and got some results. I wonder if we can have a summary section where it shows x / Y test passed, failure: openapi_version_available False, mcp_endpoint False so users can easily see the gaps:

(base) ➜  OpenEnv git:(validate-in-cli) ✗ openenv validate https://openenv-wordle.hf.space

{
  "target": "https://openenv-wordle.hf.space",
  "validation_type": "running_environment",
  "standard_version": "1.0.0",
  "standard_profile": "openenv-http/1.x",
  "mode": "simulation",
  "passed": false,
  "criteria": [
    {
      "id": "openapi_version_available",
      "description": "GET /openapi.json returns OpenAPI info.version",
      "passed": true,
      "required": true,
      "expected": {
        "status_code": 200,
        "info.version": "string"
      },
      "actual": {
        "status_code": 200,
        "info.version": "1.0.0"
      }
    },
    {
      "id": "health_endpoint",
      "description": "GET /health returns healthy status",
      "passed": true,
      "required": true,
      "expected": {
        "status_code": 200,
        "status": "healthy"
      },
      "actual": {
        "status_code": 200,
        "status": "healthy"
      }
    },
    {
      "id": "metadata_endpoint",
      "description": "GET /metadata returns name and description",
      "passed": true,
      "required": true,
      "expected": {
        "status_code": 200,
        "fields": [
          "name",
          "description"
        ]
      },
      "actual": {
        "status_code": 200,
        "name": "TextArenaEnvironment",
        "description": "TextArenaEnvironment environment"
      }
    },
    {
      "id": "schema_endpoint",
      "description": "GET /schema returns action, observation, and state schemas",
      "passed": true,
      "required": true,
      "expected": {
        "status_code": 200,
        "fields": [
          "action",
          "observation",
          "state"
        ]
      },
      "actual": {
        "status_code": 200,
        "has_action": true,
        "has_observation": true,
        "has_state": true
      }
    },
    {
      "id": "mcp_endpoint",
      "description": "POST /mcp is reachable and returns JSON-RPC payload",
      "passed": false,
      "required": true,
      "expected": {
        "status_code": 200,
        "jsonrpc": "2.0"
      },
      "actual": {
        "status_code": 404,
        "jsonrpc": null
      }
    },
    {
      "id": "mode_endpoint_consistency",
      "description": "OpenAPI endpoint set matches OpenEnv mode contract",
      "passed": true,
      "required": true,
      "expected": {
        "/reset": true,
        "/step": true,
        "/state": true
      },
      "actual": {
        "/reset": true,
        "/step": true,
        "/state": true
      }
    }
  ]
}
(base) ➜  OpenEnv git:(validate-in-cli) ✗ openenv validate https://huggingface.co/spaces/wukaixingxp/coding-env-test

{
  "target": "https://huggingface.co/spaces/wukaixingxp/coding-env-test",
  "validation_type": "running_environment",
  "standard_version": "unknown",
  "standard_profile": "openenv-http/unknown",
  "mode": "unknown",
  "passed": false,
  "criteria": [
    {
      "id": "openapi_version_available",
      "description": "GET /openapi.json returns OpenAPI info.version",
      "passed": false,
      "required": true,
      "details": "Response missing required OpenAPI info.version field",
      "expected": {
        "status_code": 200,
        "info.version": "string"
      },
      "actual": {
        "status_code": 400,
        "body_type": "non_json"
      }
    },
    {
      "id": "health_endpoint",
      "description": "GET /health returns healthy status",
      "passed": false,
      "required": true,
      "expected": {
        "status_code": 200,
        "status": "healthy"
      },
      "actual": {
        "status_code": 404,
        "status": null
      }
    },
    {
      "id": "metadata_endpoint",
      "description": "GET /metadata returns name and description",
      "passed": false,
      "required": true,
      "expected": {
        "status_code": 200,
        "fields": [
          "name",
          "description"
        ]
      },
      "actual": {
        "status_code": 404,
        "name": null,
        "description": null
      }
    },
    {
      "id": "schema_endpoint",
      "description": "GET /schema returns action, observation, and state schemas",
      "passed": false,
      "required": true,
      "expected": {
        "status_code": 200,
        "fields": [
          "action",
          "observation",
          "state"
        ]
      },
      "actual": {
        "status_code": 404,
        "has_action": false,
        "has_observation": false,
        "has_state": false
      }
    },
    {
      "id": "mcp_endpoint",
      "description": "POST /mcp is reachable and returns JSON-RPC payload",
      "passed": false,
      "required": true,
      "expected": {
        "status_code": 200,
        "jsonrpc": "2.0"
      },
      "actual": {
        "status_code": 404,
        "jsonrpc": null
      }
    },
    {
      "id": "mode_endpoint_consistency",
      "description": "OpenAPI endpoint set matches OpenEnv mode contract",
      "passed": false,
      "required": true,
      "details": "Cannot determine mode without OpenAPI paths",
      "expected": {
        "openapi.paths": "present"
      },
      "actual": {
        "openapi.paths": "missing"
      }
    }
  ]
}