[Phase 2.5-F] Harden verification gates and add batch-level QA

[frankbria]

Context

Re-running the end-to-end CLI validation (#353) with the ReAct agent revealed that all 12 tasks completed successfully (DONE status, 0 blockers, 18 min total), but the generated code has 3 categories of bugs that the verification gates failed to catch. The orchestration pipeline works; the quality assurance has gaps.

Test run: cf-test project, todo app (FastAPI + Next.js), serial strategy, 12 LLM-generated tasks.

Bugs the gates missed

Bug	Root Cause	Why Gate Missed It
init_db() signature mismatch — test passes arg, function takes 0	Agent wrote tests against an imagined API	pytest gate likely saw import errors and treated as "no tests" rather than failure
Double /api/api/ prefix — APIRouter(prefix="/api") + include_router(prefix="/api")	Cross-file inconsistency between tasks	No integration/smoke test gate exists
TypeScript type error — completed passed to CreateTodoRequest which only has title	Frontend tasks didn't cross-check the API service types	No tsc gate exists in the gate system
Task ordering — requirements.txt generated as task 11 but tests ran in task 6	Serial strategy uses priority order, not dependency order	No dep-install-before-test enforcement

Requirements

1. Add TypeScript type-check gate

• Detect tsconfig.json in workspace → run npx tsc --noEmit or npm run type-check if script exists
• Add to _detect_available_gates() in gates.py
• Timeout: 2 min (same as mypy)
• This would have caught the CreateTodoRequest type error

2. Harden pytest gate failure handling

• If pytest returns non-zero due to import errors or collection errors, the gate must report FAILED (not skip)
• Distinguish between "no tests found" (acceptable) vs "tests errored during collection" (failure)
• Parse pytest output for ERROR vs no tests ran patterns

3. Add build verification gate

• Python: python -c "from main import app" (or detect entry point)
• Node/TS: npm run build if build script exists in package.json
• This catches import errors, missing deps, and type errors in one shot
• Run as part of final verification, not incremental

4. Add end-of-batch validation step

• After BATCH_COMPLETED in conductor.py, re-run all gates against the full workspace
• This catches cross-task inconsistencies that per-task gates miss
• Report batch as PARTIAL if post-batch gates fail (tasks DONE but integration broken)
• Hook point: after the final BATCH_COMPLETED event emission

5. Ensure dependency installation before test gates

• Before running pytest/npm-test gates, check if dependencies are installed
• If requirements.txt exists but no venv/site-packages → run pip install -r requirements.txt
• If package.json exists but no node_modules/ → run npm install
• This is a prerequisite step in the gate runner, not a separate gate

Acceptance Criteria

• tsc --noEmit gate auto-detected and run for TypeScript projects
• pytest gate fails on collection/import errors (not silent skip)
• Build gate (npm run build / Python import check) runs during final verification
• Conductor runs full gate sweep after batch completion
• Gate runner installs missing deps before running test gates
• Re-run the cf-test todo app scenario — all 3 bug categories either caught or prevented

Ordering

This issue must be completed before:

• [Phase 2.5-F] Verify ReAct engine works via API routes #354 — Verify ReAct engine works via API routes
• [Phase 2.5-F] Switch default engine to react and update documentation #355 — Switch default engine to react and update documentation

The gate hardening ensures that the validation in #354 is meaningful and the engine switch in #355 is backed by real quality checks.

Test Plan

1. Run the same cf-test todo app scenario after implementing fixes
2. Verify that the agent either:
   • Produces correct code (gates prevent the bugs during self-correction), OR
   • Gates catch the bugs and tasks are marked FAILED (not false DONE)
3. Verify batch-level validation catches cross-task inconsistencies

[coderabbitai]

🔗 Related PRs

#50 - feat: Implement Quality Gates Panel in Dashboard (#43) [merged]
#303 - feat(gates): add BUILD validation gate for config errors [merged]
#363 - feat: wire ReactAgent into runtime with --engine react CLI flag [merged]
#369 - test: add CLI integration tests for ReactAgent runtime parameters [merged]
#384 - feat: adaptive iteration budget, autofix, and loop detection for ReactAgent [closed]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[traycerai]

Plan

Observations

The gate system in file:codeframe/core/gates.py currently supports pytest, ruff, mypy, npm-test, and npm-lint gates. The pytest gate (line 287-345) treats all non-zero exit codes as failures without distinguishing between collection errors and actual test failures. No TypeScript type-check gate exists, and there's no build verification gate. The conductor emits BATCH_COMPLETED events in four locations (_execute_serial, _execute_parallel, _execute_retries, resume_batch) without post-batch validation. The quality_gates.py module has more comprehensive gates but appears to be a separate system.

Approach

This plan implements five hardening improvements to the verification gate system: (1) Add TypeScript type-check gate detection and execution, (2) Enhance pytest gate to fail on collection/import errors rather than silently skipping, (3) Add build verification gates for Python and Node.js projects, (4) Implement batch-level validation that re-runs all gates after BATCH_COMPLETED, and (5) Add dependency installation prerequisite checks before running test gates. All changes will be made to file:codeframe/core/gates.py and file:codeframe/core/conductor.py to maintain consistency with the existing gate architecture.

Implementation Steps

1. Add TypeScript Type-Check Gate

File: file:codeframe/core/gates.py

Update _detect_available_gates() function (line 250-283):

• After the npm-lint detection block (line 278-281), add TypeScript detection logic
• Check if tsconfig.json exists in repo_path
• If found, append "tsc" to the gates list
• This enables auto-detection for TypeScript projects

Update the known_gates set in run() function (line 190):

• Add "tsc" to the set: {"pytest", "ruff", "mypy", "npm-test", "npm-lint", "tsc"}

Add gate execution handler in run() function (line 193-218):

• Add new elif branch after npm-lint check (after line 203)
• Call _run_tsc(repo_path, verbose) when gate_name is "tsc"

Create new _run_tsc() function (add after _run_npm_lint() around line 559):

• Check if npx is available using shutil.which("npx")
• Return SKIPPED if not found
• Check if tsconfig.json exists, return SKIPPED if missing
• Parse package.json to check if type-check script exists
• If script exists, run npm run type-check
• Otherwise, run npx tsc --noEmit
• Set timeout to 120 seconds (2 minutes, same as mypy)
• Capture stdout/stderr and return GateCheck with PASSED/FAILED status
• Truncate output to 2000 chars if not verbose

2. Harden Pytest Gate Failure Handling

File: file:codeframe/core/gates.py

Enhance _run_pytest() function (line 287-345):

• After capturing output (line 318-320), add collection error detection
• Check for pytest-specific error patterns in output:
  • "ERROR collecting" - collection phase errors
  • "ImportError" or "ModuleNotFoundError" - import failures
  • "INTERNALERROR" - pytest internal errors
• If non-zero exit code AND error patterns found, return FAILED status
• If exit code is 5 (pytest's "no tests collected" code) AND output contains "no tests ran", return PASSED (acceptable)
• Otherwise, use existing logic (line 326-332)

Add helper function _has_pytest_collection_errors() (add after _summarize_pytest_output() around line 835):

• Accept output string parameter
• Use regex to detect collection/import errors
• Return tuple: (has_errors: bool, error_type: str)
• Error types: "collection", "import", "internal", or None

Update _summarize_pytest_output() function (line 819-834):

• Add check for error patterns before looking for summary line
• If collection errors detected, extract and return first error line
• Preserve existing summary extraction logic for normal test runs

3. Add Build Verification Gate

File: file:codeframe/core/gates.py

Update _detect_available_gates() function (line 250-283):

• Add build gate detection at the end (after TypeScript detection)
• For Python: if pyproject.toml or setup.py exists, append "python-build"
• For Node.js: if package.json exists and has "build" script, append "npm-build"

Update known_gates set in run() function (line 190):

• Add "python-build" and "npm-build" to the set

Add gate execution handlers in run() function (line 193-218):

• Add elif branch for "python-build": call _run_python_build(repo_path, verbose)
• Add elif branch for "npm-build": call _run_npm_build(repo_path, verbose)

Create _run_python_build() function (add after _run_tsc() around line 600):

• Detect entry point: check for main.py, app.py, or parse pyproject.toml for entry point
• If found, run python -c "from <module> import <app>" to verify imports
• Use uv run python if uv is available
• Timeout: 60 seconds
• Return PASSED if exit code 0, FAILED otherwise
• Capture import errors in output

Create _run_npm_build() function (add after _run_python_build() around line 640):

• Check if npm is available
• Verify package.json has "build" script
• Run npm run build
• Timeout: 300 seconds (5 minutes for builds)
• Return PASSED if exit code 0, FAILED otherwise
• Truncate output to 5000 chars if not verbose

4. Add End-of-Batch Validation

File: file:codeframe/core/conductor.py

Create new function _run_batch_level_validation() (add after _execute_retries() around line 1147):

• Accept parameters: workspace: Workspace, batch: BatchRun
• Import gates module: from codeframe.core import gates
• Run gates.run(workspace, gates=None, verbose=False) to execute all detected gates
• Check if result.passed is False
• If failed, update batch status to PARTIAL (tasks done but integration broken)
• Emit event events.EventType.BATCH_VALIDATION_FAILED with gate failure details
• Return validation result for logging

Update _execute_serial() function (line 1148-1285):

• After determining final batch status (line 1248-1260), before setting completed_at
• If completed_count == total (all tasks completed), call _run_batch_level_validation()
• If validation fails, change batch.status to BatchStatus.PARTIAL
• Change event_type to events.EventType.BATCH_PARTIAL
• Add validation failure info to event payload

Update _execute_parallel() function (line 1287-1416):

• After determining final batch status (line 1377-1388), before setting completed_at
• If completed_count == total, call _run_batch_level_validation()
• Apply same status change logic as serial execution

Update _execute_retries() function (line 1013-1146):

• After final status calculation (line 1110-1121), before setting completed_at
• If final_completed == total, call _run_batch_level_validation()
• Apply same status change logic

Update resume_batch() function (line 788-1011):

• After final status calculation (line 974-985), before setting completed_at
• If final_completed == total, call _run_batch_level_validation()
• Apply same status change logic

File: file:codeframe/core/events.py

Add new event type (around line 104):

• BATCH_VALIDATION_FAILED = "BATCH_VALIDATION_FAILED"
• This event signals that batch-level gates failed after all tasks completed

5. Ensure Dependency Installation Before Test Gates

File: file:codeframe/core/gates.py

Create helper function _ensure_dependencies_installed() (add before _run_pytest() around line 285):

• Accept repo_path: Path parameter
• Check Python dependencies:
  • If requirements.txt exists but no .venv/ or venv/ directory
  • Run uv pip install -r requirements.txt if uv available, else pip install -r requirements.txt
  • Timeout: 300 seconds
  • Log installation attempt
• Check Node.js dependencies:
  • If package.json exists but no node_modules/ directory
  • Run npm install
  • Timeout: 300 seconds
  • Log installation attempt
• Return tuple: (success: bool, message: str)
• Catch exceptions and return (False, error_message)

Update _run_pytest() function (line 287-345):

• Before running pytest command (line 302), call _ensure_dependencies_installed(repo_path)
• If installation fails, return GateCheck with ERROR status and installation error message
• Continue with existing pytest execution if successful

Update _run_npm_test() function (line 461-508):

• Before running npm test (line 474), call _ensure_dependencies_installed(repo_path)
• If installation fails, return GateCheck with ERROR status
• Continue with existing npm test execution if successful

Verification Flow

sequenceDiagram
    participant Conductor
    participant Gates
    participant Workspace
    
    Note over Conductor: Task Execution Phase
    Conductor->>Conductor: Execute tasks (serial/parallel)
    Conductor->>Conductor: All tasks COMPLETED
    
    Note over Conductor: Batch-Level Validation
    Conductor->>Gates: run_batch_level_validation()
    Gates->>Gates: _ensure_dependencies_installed()
    Gates->>Workspace: Install missing deps
    Gates->>Gates: _detect_available_gates()
    Gates->>Gates: Run pytest (with collection error check)
    Gates->>Gates: Run tsc --noEmit
    Gates->>Gates: Run python-build
    Gates->>Gates: Run npm-build
    Gates->>Gates: Run ruff, mypy, etc.
    Gates-->>Conductor: GateResult (passed/failed)
    
    alt Gates Failed
        Conductor->>Conductor: Change status to PARTIAL
        Conductor->>Workspace: Emit BATCH_VALIDATION_FAILED
    else Gates Passed
        Conductor->>Conductor: Keep status COMPLETED
        Conductor->>Workspace: Emit BATCH_COMPLETED
    end

Loading

Expected Bug Detection

Bug Category	Detection Mechanism	Gate Responsible
init_db() signature mismatch	Pytest collection error detection	Enhanced pytest gate
Double /api/api/ prefix	Integration test failure or build error	Batch-level validation + build gate
TypeScript type error (completed field)	Type checking	New tsc gate
Missing dependencies before tests	Dependency installation check	_ensure_dependencies_installed()

Import In IDE

🤖 Prompt for AI Agents

## Observations

The gate system in `file:codeframe/core/gates.py` currently supports pytest, ruff, mypy, npm-test, and npm-lint gates. The pytest gate (line 287-345) treats all non-zero exit codes as failures without distinguishing between collection errors and actual test failures. No TypeScript type-check gate exists, and there's no build verification gate. The conductor emits BATCH_COMPLETED events in four locations (_execute_serial, _execute_parallel, _execute_retries, resume_batch) without post-batch validation. The quality_gates.py module has more comprehensive gates but appears to be a separate system.

## Approach

This plan implements five hardening improvements to the verification gate system: (1) Add TypeScript type-check gate detection and execution, (2) Enhance pytest gate to fail on collection/import errors rather than silently skipping, (3) Add build verification gates for Python and Node.js projects, (4) Implement batch-level validation that re-runs all gates after BATCH_COMPLETED, and (5) Add dependency installation prerequisite checks before running test gates. All changes will be made to `file:codeframe/core/gates.py` and `file:codeframe/core/conductor.py` to maintain consistency with the existing gate architecture.

## Implementation Steps

### 1. Add TypeScript Type-Check Gate

**File: `file:codeframe/core/gates.py`**

Update `_detect_available_gates()` function (line 250-283):
- After the npm-lint detection block (line 278-281), add TypeScript detection logic
- Check if `tsconfig.json` exists in `repo_path`
- If found, append `"tsc"` to the gates list
- This enables auto-detection for TypeScript projects

Update the `known_gates` set in `run()` function (line 190):
- Add `"tsc"` to the set: `{"pytest", "ruff", "mypy", "npm-test", "npm-lint", "tsc"}`

Add gate execution handler in `run()` function (line 193-218):
- Add new elif branch after `npm-lint` check (after line 203)
- Call `_run_tsc(repo_path, verbose)` when gate_name is "tsc"

Create new `_run_tsc()` function (add after `_run_npm_lint()` around line 559):
- Check if `npx` is available using `shutil.which("npx")`
- Return SKIPPED if not found
- Check if `tsconfig.json` exists, return SKIPPED if missing
- Parse `package.json` to check if `type-check` script exists
- If script exists, run `npm run type-check`
- Otherwise, run `npx tsc --noEmit`
- Set timeout to 120 seconds (2 minutes, same as mypy)
- Capture stdout/stderr and return GateCheck with PASSED/FAILED status
- Truncate output to 2000 chars if not verbose

### 2. Harden Pytest Gate Failure Handling

**File: `file:codeframe/core/gates.py`**

Enhance `_run_pytest()` function (line 287-345):
- After capturing output (line 318-320), add collection error detection
- Check for pytest-specific error patterns in output:
  - `"ERROR collecting"` - collection phase errors
  - `"ImportError"` or `"ModuleNotFoundError"` - import failures
  - `"INTERNALERROR"` - pytest internal errors
- If non-zero exit code AND error patterns found, return FAILED status
- If exit code is 5 (pytest's "no tests collected" code) AND output contains "no tests ran", return PASSED (acceptable)
- Otherwise, use existing logic (line 326-332)

Add helper function `_has_pytest_collection_errors()` (add after `_summarize_pytest_output()` around line 835):
- Accept output string parameter
- Use regex to detect collection/import errors
- Return tuple: (has_errors: bool, error_type: str)
- Error types: "collection", "import", "internal", or None

Update `_summarize_pytest_output()` function (line 819-834):
- Add check for error patterns before looking for summary line
- If collection errors detected, extract and return first error line
- Preserve existing summary extraction logic for normal test runs

### 3. Add Build Verification Gate

**File: `file:codeframe/core/gates.py`**

Update `_detect_available_gates()` function (line 250-283):
- Add build gate detection at the end (after TypeScript detection)
- For Python: if `pyproject.toml` or `setup.py` exists, append `"python-build"`
- For Node.js: if `package.json` exists and has `"build"` script, append `"npm-build"`

Update `known_gates` set in `run()` function (line 190):
- Add `"python-build"` and `"npm-build"` to the set

Add gate execution handlers in `run()` function (line 193-218):
- Add elif branch for `"python-build"`: call `_run_python_build(repo_path, verbose)`
- Add elif branch for `"npm-build"`: call `_run_npm_build(repo_path, verbose)`

Create `_run_python_build()` function (add after `_run_tsc()` around line 600):
- Detect entry point: check for `main.py`, `app.py`, or parse `pyproject.toml` for entry point
- If found, run `python -c "from <module> import <app>"` to verify imports
- Use `uv run python` if uv is available
- Timeout: 60 seconds
- Return PASSED if exit code 0, FAILED otherwise
- Capture import errors in output

Create `_run_npm_build()` function (add after `_run_python_build()` around line 640):
- Check if `npm` is available
- Verify `package.json` has `"build"` script
- Run `npm run build`
- Timeout: 300 seconds (5 minutes for builds)
- Return PASSED if exit code 0, FAILED otherwise
- Truncate output to 5000 chars if not verbose

### 4. Add End-of-Batch Validation

**File: `file:codeframe/core/conductor.py`**

Create new function `_run_batch_level_validation()` (add after `_execute_retries()` around line 1147):
- Accept parameters: `workspace: Workspace`, `batch: BatchRun`
- Import gates module: `from codeframe.core import gates`
- Run `gates.run(workspace, gates=None, verbose=False)` to execute all detected gates
- Check if `result.passed` is False
- If failed, update batch status to PARTIAL (tasks done but integration broken)
- Emit event `events.EventType.BATCH_VALIDATION_FAILED` with gate failure details
- Return validation result for logging

Update `_execute_serial()` function (line 1148-1285):
- After determining final batch status (line 1248-1260), before setting `completed_at`
- If `completed_count == total` (all tasks completed), call `_run_batch_level_validation()`
- If validation fails, change `batch.status` to `BatchStatus.PARTIAL`
- Change `event_type` to `events.EventType.BATCH_PARTIAL`
- Add validation failure info to event payload

Update `_execute_parallel()` function (line 1287-1416):
- After determining final batch status (line 1377-1388), before setting `completed_at`
- If `completed_count == total`, call `_run_batch_level_validation()`
- Apply same status change logic as serial execution

Update `_execute_retries()` function (line 1013-1146):
- After final status calculation (line 1110-1121), before setting `completed_at`
- If `final_completed == total`, call `_run_batch_level_validation()`
- Apply same status change logic

Update `resume_batch()` function (line 788-1011):
- After final status calculation (line 974-985), before setting `completed_at`
- If `final_completed == total`, call `_run_batch_level_validation()`
- Apply same status change logic

**File: `file:codeframe/core/events.py`**

Add new event type (around line 104):
- `BATCH_VALIDATION_FAILED = "BATCH_VALIDATION_FAILED"`
- This event signals that batch-level gates failed after all tasks completed

### 5. Ensure Dependency Installation Before Test Gates

**File: `file:codeframe/core/gates.py`**

Create helper function `_ensure_dependencies_installed()` (add before `_run_pytest()` around line 285):
- Accept `repo_path: Path` parameter
- Check Python dependencies:
  - If `requirements.txt` exists but no `.venv/` or `venv/` directory
  - Run `uv pip install -r requirements.txt` if uv available, else `pip install -r requirements.txt`
  - Timeout: 300 seconds
  - Log installation attempt
- Check Node.js dependencies:
  - If `package.json` exists but no `node_modules/` directory
  - Run `npm install`
  - Timeout: 300 seconds
  - Log installation attempt
- Return tuple: (success: bool, message: str)
- Catch exceptions and return (False, error_message)

Update `_run_pytest()` function (line 287-345):
- Before running pytest command (line 302), call `_ensure_dependencies_installed(repo_path)`
- If installation fails, return GateCheck with ERROR status and installation error message
- Continue with existing pytest execution if successful

Update `_run_npm_test()` function (line 461-508):
- Before running npm test (line 474), call `_ensure_dependencies_installed(repo_path)`
- If installation fails, return GateCheck with ERROR status
- Continue with existing npm test execution if successful

## Verification Flow

```mermaid
sequenceDiagram
    participant Conductor
    participant Gates
    participant Workspace
    
    Note over Conductor: Task Execution Phase
    Conductor->>Conductor: Execute tasks (serial/parallel)
    Conductor->>Conductor: All tasks COMPLETED
    
    Note over Conductor: Batch-Level Validation
    Conductor->>Gates: run_batch_level_validation()
    Gates->>Gates: _ensure_dependencies_installed()
    Gates->>Workspace: Install missing deps
    Gates->>Gates: _detect_available_gates()
    Gates->>Gates: Run pytest (with collection error check)
    Gates->>Gates: Run tsc --noEmit
    Gates->>Gates: Run python-build
    Gates->>Gates: Run npm-build
    Gates->>Gates: Run ruff, mypy, etc.
    Gates-->>Conductor: GateResult (passed/failed)
    
    alt Gates Failed
        Conductor->>Conductor: Change status to PARTIAL
        Conductor->>Workspace: Emit BATCH_VALIDATION_FAILED
    else Gates Passed
        Conductor->>Conductor: Keep status COMPLETED
        Conductor->>Workspace: Emit BATCH_COMPLETED
    end
```

## Expected Bug Detection

| Bug Category | Detection Mechanism | Gate Responsible |
|--------------|---------------------|------------------|
| `init_db()` signature mismatch | Pytest collection error detection | Enhanced pytest gate |
| Double `/api/api/` prefix | Integration test failure or build error | Batch-level validation + build gate |
| TypeScript type error (`completed` field) | Type checking | New tsc gate |
| Missing dependencies before tests | Dependency installation check | _ensure_dependencies_installed() |

---

Execution Information

Branch: main
Commit: fe160c6

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.