[Phase 5] Debug and replay mode

[frankbria]

Summary

Enable stepping through past executions for debugging and learning.

Problem

When agent execution fails or produces unexpected results, it's hard to understand what happened. Need ability to replay and inspect past runs.

Proposed Solution

Execution Recording

• Record all LLM prompts/responses
• Track file operations with before/after state
• Log decision points and branching

Replay Mode

cf work replay <run-id>            # Step through execution
cf work replay <run-id> --step N   # Jump to step N
cf work diff <run-id>              # Show all file changes

Debug Features

• Step forward/backward through execution
• Inspect context at each step
• Compare expected vs actual outputs
• "What-if" re-execution with modified inputs

Acceptance Criteria

• Execution details recorded to database
• Replay command shows step-by-step execution
• File diffs available at each step
• Execution can be re-run from any checkpoint
• Export execution trace for analysis

Related

• Useful for debugging self-correction loop
• Enables learning from successful executions

[traycerai]

Plan

Observations

The codebase already has substantial infrastructure for execution tracking:

• Run tracking: file:codeframe/core/runtime.py manages execution runs with status, timestamps, and lifecycle
• Structured logging: file:codeframe/core/diagnostics.py provides RunLogger with categorized log entries stored in run_logs table
• File change tracking: file:codeframe/core/executor.py tracks FileChange objects with before/after content
• Checkpoint system: Database has checkpoint support via file:codeframe/persistence/repositories/checkpoint_repository.py
• CLI framework: file:codeframe/cli/app.py uses Typer with existing work_app sub-commands
• Database schema: SQLite with tables for runs, run_logs, diagnostic_reports, checkpoints

The foundation exists but needs enhancement to capture LLM interactions, decision points, and enable step-by-step replay.

Approach

Extend the existing execution recording infrastructure to capture complete execution traces, then build replay commands that allow stepping through past runs. The implementation will:

1. Enhance execution recording by extending RunLogger to capture LLM prompts/responses, decision points, and file operation details
2. Create new database tables for execution steps and LLM interactions
3. Build replay commands (cf work replay, cf work diff) that reconstruct execution state at any step
4. Implement step navigation with forward/backward stepping and jump-to-step capabilities
5. Add export functionality for execution trace analysis

This approach leverages existing infrastructure (RunLogger, FileChange tracking, checkpoint system) while adding minimal new components focused on replay capabilities.

---

Implementation Steps

1. Database Schema Extensions

Create execution steps table to track individual execution steps with their context:

Add to file:codeframe/core/workspace.py schema initialization:

• Table execution_steps with columns: id, run_id, step_number, step_type (planning/execution/verification/gate), description, started_at, completed_at, status, input_context, output_result, metadata (JSON)
• Table llm_interactions with columns: id, run_id, step_id, prompt, response, model, tokens_used, timestamp, purpose (planning/execution/review)
• Table file_operations with columns: id, run_id, step_id, operation_type (create/edit/delete), file_path, content_before, content_after, timestamp

Index strategy: Add indexes on run_id, step_number for efficient step retrieval during replay.

---

2. Enhanced Execution Recording

Extend RunLogger in file:codeframe/core/diagnostics.py:

Add methods:

• log_execution_step(step_number, step_type, description, input_context) - Record step start
• complete_execution_step(step_number, status, output_result) - Record step completion
• log_llm_interaction(step_id, prompt, response, model, tokens, purpose) - Capture LLM calls
• log_file_operation(step_id, operation, path, before, after) - Record file changes

Modify Executor in file:codeframe/core/executor.py:

Update execute_step() to:

• Call run_logger.log_execution_step() before execution
• Capture LLM prompts/responses in _generate_file_content() and _generate_edit_content()
• Log file operations via run_logger.log_file_operation() for each FileChange
• Call run_logger.complete_execution_step() after execution

Modify Agent in file:codeframe/core/agent.py:

Update run() method to:

• Log planning phase as execution step with plan details
• Log each verification gate as execution step
• Capture decision points (blocker creation, retry logic, supervisor interventions)

---

3. Replay Command Implementation

Create replay module at file:codeframe/core/replay.py:

Classes:

• ExecutionSnapshot - Represents state at a specific step (files, context, logs)
• ReplaySession - Manages stepping through execution history

Functions:

• load_execution_trace(workspace, run_id) - Load all steps for a run
• get_step_snapshot(workspace, run_id, step_number) - Reconstruct state at step N
• get_file_state_at_step(workspace, run_id, step_number, file_path) - Get file content at specific step
• compare_steps(workspace, run_id, step_a, step_b) - Diff between two steps

Add CLI commands to file:codeframe/cli/app.py work_app:

@work_app.command("replay")
def work_replay(
    run_id: str,
    step: Optional[int] = None,
    interactive: bool = True,
    show_llm: bool = False,
    show_files: bool = True
)

Features:

• If --step N provided: Jump to step N and display state
• If interactive mode: Show step-by-step navigation (n=next, p=previous, j=jump, q=quit)
• Display: step description, LLM interactions (if --show-llm), file changes, logs
• Use Rich library for formatted output with syntax highlighting

@work_app.command("diff")
def work_diff(
    run_id: str,
    step_from: Optional[int] = None,
    step_to: Optional[int] = None,
    files_only: bool = False
)

Features:

• Show all file changes across execution (or between specific steps)
• Display unified diff format with syntax highlighting
• Option to filter by file pattern
• Summary statistics (files created/modified/deleted)

---

4. Step Navigation & Inspection

Interactive replay interface in file:codeframe/core/replay.py:

Implement ReplaySession.interactive_mode():

• Display current step with context (description, timestamp, status)
• Show file changes at this step with diff view
• Display LLM prompt/response if available
• Show relevant log entries for this step
• Navigation commands: n (next), p (previous), j N (jump to step N), l (list all steps), f (show files), q (quit)

Context inspection features:

• show_step_context(step) - Display full input context for the step
• show_llm_prompt(step) - Show exact prompt sent to LLM
• show_llm_response(step) - Show LLM response
• show_file_diff(step, file_path) - Show before/after for specific file

---

5. Checkpoint Integration

Link checkpoints to execution steps in file:codeframe/persistence/repositories/checkpoint_repository.py:

Modify save_checkpoint() to:

• Record current step_number in checkpoint metadata
• Store execution trace summary (steps completed, files modified)

Restore from checkpoint functionality:

• restore_to_step(workspace, run_id, step_number) - Restore workspace to specific step
• Reconstruct file system state by replaying file operations up to step N
• Restore database state from nearest checkpoint before step N

---

6. Export & Analysis

Add export command to file:codeframe/cli/app.py:

@work_app.command("export-trace")
def work_export_trace(
    run_id: str,
    output_path: Path,
    format: str = "json"  # json, markdown, html
)

Export formats:

• JSON: Complete execution trace with all steps, LLM interactions, file changes
• Markdown: Human-readable report with timeline, decisions, outcomes
• HTML: Interactive visualization with collapsible sections, syntax highlighting

Export structure (JSON):

{
  "run_id": "...",
  "task_id": "...",
  "started_at": "...",
  "completed_at": "...",
  "status": "...",
  "steps": [
    {
      "step_number": 1,
      "type": "planning",
      "description": "...",
      "llm_interactions": [...],
      "file_operations": [...],
      "logs": [...]
    }
  ],
  "summary": {
    "total_steps": 10,
    "files_modified": 5,
    "llm_calls": 8,
    "total_tokens": 15000
  }
}

---

7. What-If Re-execution

Add re-execution capability in file:codeframe/core/replay.py:

Function rerun_from_step(workspace, run_id, step_number, modifications):

• Restore state to step N
• Apply user modifications (updated prompt, different parameters)
• Create new run branching from original
• Execute remaining steps with modifications
• Compare outcomes with original execution

CLI command:

@work_app.command("rerun")
def work_rerun(
    run_id: str,
    from_step: int,
    modify_prompt: Optional[str] = None,
    dry_run: bool = True
)

---

8. Testing & Validation

Unit tests to create:

• tests/core/test_replay.py - Test execution trace loading, step navigation, state reconstruction
• tests/cli/test_replay_commands.py - Test CLI commands for replay, diff, export
• tests/core/test_execution_recording.py - Test enhanced logging of steps, LLM interactions, file operations

Integration tests:

• End-to-end test: Execute task → Record trace → Replay execution → Verify state matches
• Test checkpoint restoration to specific step
• Test export in all formats (JSON, Markdown, HTML)

---

Sequence Diagram

sequenceDiagram
    participant User
    participant CLI
    participant ReplaySession
    participant Database
    participant FileSystem

    User->>CLI: cf work replay <run-id>
    CLI->>Database: Load execution trace
    Database-->>CLI: Steps, LLM interactions, file ops
    CLI->>ReplaySession: Create session
    
    loop Interactive Navigation
        ReplaySession->>Database: Get step N details
        Database-->>ReplaySession: Step data
        ReplaySession->>FileSystem: Reconstruct file state
        FileSystem-->>ReplaySession: File contents
        ReplaySession->>User: Display step (diff, logs, LLM)
        User->>ReplaySession: Navigate (n/p/j/q)
    end
    
    User->>CLI: cf work diff <run-id>
    CLI->>Database: Load file operations
    Database-->>CLI: All file changes
    CLI->>User: Display unified diff
    
    User->>CLI: cf work export-trace <run-id>
    CLI->>Database: Load complete trace
    Database-->>CLI: Full execution data
    CLI->>FileSystem: Write export file
    CLI->>User: Export complete

Loading

---

File References

Files to modify:

• file:codeframe/core/workspace.py - Add new tables for execution_steps, llm_interactions, file_operations
• file:codeframe/core/diagnostics.py - Extend RunLogger with step/LLM/file operation logging
• file:codeframe/core/executor.py - Add execution recording hooks
• file:codeframe/core/agent.py - Add step logging for planning and verification
• file:codeframe/cli/app.py - Add replay, diff, export-trace, rerun commands
• file:codeframe/persistence/repositories/checkpoint_repository.py - Link checkpoints to steps

Files to create:

• file:codeframe/core/replay.py - Replay session management and state reconstruction
• file:tests/core/test_replay.py - Unit tests for replay functionality
• file:tests/cli/test_replay_commands.py - CLI command tests

Existing infrastructure to leverage:

• file:codeframe/core/runtime.py - Run lifecycle management
• file:codeframe/core/diagnostics.py - RunLogger and log storage
• file:codeframe/core/executor.py - FileChange tracking
• file:codeframe/persistence/database.py - Database access layer

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase already has substantial infrastructure for execution tracking:
- **Run tracking**: file:codeframe/core/runtime.py manages execution runs with status, timestamps, and lifecycle
- **Structured logging**: file:codeframe/core/diagnostics.py provides `RunLogger` with categorized log entries stored in `run_logs` table
- **File change tracking**: file:codeframe/core/executor.py tracks `FileChange` objects with before/after content
- **Checkpoint system**: Database has checkpoint support via file:codeframe/persistence/repositories/checkpoint_repository.py
- **CLI framework**: file:codeframe/cli/app.py uses Typer with existing `work_app` sub-commands
- **Database schema**: SQLite with tables for runs, run_logs, diagnostic_reports, checkpoints

The foundation exists but needs enhancement to capture LLM interactions, decision points, and enable step-by-step replay.

## Approach

Extend the existing execution recording infrastructure to capture complete execution traces, then build replay commands that allow stepping through past runs. The implementation will:

1. **Enhance execution recording** by extending `RunLogger` to capture LLM prompts/responses, decision points, and file operation details
2. **Create new database tables** for execution steps and LLM interactions
3. **Build replay commands** (`cf work replay`, `cf work diff`) that reconstruct execution state at any step
4. **Implement step navigation** with forward/backward stepping and jump-to-step capabilities
5. **Add export functionality** for execution trace analysis

This approach leverages existing infrastructure (RunLogger, FileChange tracking, checkpoint system) while adding minimal new components focused on replay capabilities.

---

## Implementation Steps

### 1. Database Schema Extensions

**Create execution steps table** to track individual execution steps with their context:

Add to file:codeframe/core/workspace.py schema initialization:
- Table `execution_steps` with columns: id, run_id, step_number, step_type (planning/execution/verification/gate), description, started_at, completed_at, status, input_context, output_result, metadata (JSON)
- Table `llm_interactions` with columns: id, run_id, step_id, prompt, response, model, tokens_used, timestamp, purpose (planning/execution/review)
- Table `file_operations` with columns: id, run_id, step_id, operation_type (create/edit/delete), file_path, content_before, content_after, timestamp

**Index strategy**: Add indexes on run_id, step_number for efficient step retrieval during replay.

---

### 2. Enhanced Execution Recording

**Extend RunLogger** in file:codeframe/core/diagnostics.py:

Add methods:
- `log_execution_step(step_number, step_type, description, input_context)` - Record step start
- `complete_execution_step(step_number, status, output_result)` - Record step completion
- `log_llm_interaction(step_id, prompt, response, model, tokens, purpose)` - Capture LLM calls
- `log_file_operation(step_id, operation, path, before, after)` - Record file changes

**Modify Executor** in file:codeframe/core/executor.py:

Update `execute_step()` to:
- Call `run_logger.log_execution_step()` before execution
- Capture LLM prompts/responses in `_generate_file_content()` and `_generate_edit_content()`
- Log file operations via `run_logger.log_file_operation()` for each FileChange
- Call `run_logger.complete_execution_step()` after execution

**Modify Agent** in file:codeframe/core/agent.py:

Update `run()` method to:
- Log planning phase as execution step with plan details
- Log each verification gate as execution step
- Capture decision points (blocker creation, retry logic, supervisor interventions)

---

### 3. Replay Command Implementation

**Create replay module** at file:codeframe/core/replay.py:

Classes:
- `ExecutionSnapshot` - Represents state at a specific step (files, context, logs)
- `ReplaySession` - Manages stepping through execution history

Functions:
- `load_execution_trace(workspace, run_id)` - Load all steps for a run
- `get_step_snapshot(workspace, run_id, step_number)` - Reconstruct state at step N
- `get_file_state_at_step(workspace, run_id, step_number, file_path)` - Get file content at specific step
- `compare_steps(workspace, run_id, step_a, step_b)` - Diff between two steps

**Add CLI commands** to file:codeframe/cli/app.py work_app:

```
@work_app.command("replay")
def work_replay(
    run_id: str,
    step: Optional[int] = None,
    interactive: bool = True,
    show_llm: bool = False,
    show_files: bool = True
)
```
Features:
- If `--step N` provided: Jump to step N and display state
- If interactive mode: Show step-by-step navigation (n=next, p=previous, j=jump, q=quit)
- Display: step description, LLM interactions (if --show-llm), file changes, logs
- Use Rich library for formatted output with syntax highlighting

```
@work_app.command("diff")
def work_diff(
    run_id: str,
    step_from: Optional[int] = None,
    step_to: Optional[int] = None,
    files_only: bool = False
)
```
Features:
- Show all file changes across execution (or between specific steps)
- Display unified diff format with syntax highlighting
- Option to filter by file pattern
- Summary statistics (files created/modified/deleted)

---

### 4. Step Navigation & Inspection

**Interactive replay interface** in file:codeframe/core/replay.py:

Implement `ReplaySession.interactive_mode()`:
- Display current step with context (description, timestamp, status)
- Show file changes at this step with diff view
- Display LLM prompt/response if available
- Show relevant log entries for this step
- Navigation commands: n (next), p (previous), j N (jump to step N), l (list all steps), f (show files), q (quit)

**Context inspection** features:
- `show_step_context(step)` - Display full input context for the step
- `show_llm_prompt(step)` - Show exact prompt sent to LLM
- `show_llm_response(step)` - Show LLM response
- `show_file_diff(step, file_path)` - Show before/after for specific file

---

### 5. Checkpoint Integration

**Link checkpoints to execution steps** in file:codeframe/persistence/repositories/checkpoint_repository.py:

Modify `save_checkpoint()` to:
- Record current step_number in checkpoint metadata
- Store execution trace summary (steps completed, files modified)

**Restore from checkpoint** functionality:
- `restore_to_step(workspace, run_id, step_number)` - Restore workspace to specific step
- Reconstruct file system state by replaying file operations up to step N
- Restore database state from nearest checkpoint before step N

---

### 6. Export & Analysis

**Add export command** to file:codeframe/cli/app.py:

```
@work_app.command("export-trace")
def work_export_trace(
    run_id: str,
    output_path: Path,
    format: str = "json"  # json, markdown, html
)
```

Export formats:
- **JSON**: Complete execution trace with all steps, LLM interactions, file changes
- **Markdown**: Human-readable report with timeline, decisions, outcomes
- **HTML**: Interactive visualization with collapsible sections, syntax highlighting

**Export structure** (JSON):
```json
{
  "run_id": "...",
  "task_id": "...",
  "started_at": "...",
  "completed_at": "...",
  "status": "...",
  "steps": [
    {
      "step_number": 1,
      "type": "planning",
      "description": "...",
      "llm_interactions": [...],
      "file_operations": [...],
      "logs": [...]
    }
  ],
  "summary": {
    "total_steps": 10,
    "files_modified": 5,
    "llm_calls": 8,
    "total_tokens": 15000
  }
}
```

---

### 7. What-If Re-execution

**Add re-execution capability** in file:codeframe/core/replay.py:

Function `rerun_from_step(workspace, run_id, step_number, modifications)`:
- Restore state to step N
- Apply user modifications (updated prompt, different parameters)
- Create new run branching from original
- Execute remaining steps with modifications
- Compare outcomes with original execution

**CLI command**:
```
@work_app.command("rerun")
def work_rerun(
    run_id: str,
    from_step: int,
    modify_prompt: Optional[str] = None,
    dry_run: bool = True
)
```

---

### 8. Testing & Validation

**Unit tests** to create:
- `tests/core/test_replay.py` - Test execution trace loading, step navigation, state reconstruction
- `tests/cli/test_replay_commands.py` - Test CLI commands for replay, diff, export
- `tests/core/test_execution_recording.py` - Test enhanced logging of steps, LLM interactions, file operations

**Integration tests**:
- End-to-end test: Execute task → Record trace → Replay execution → Verify state matches
- Test checkpoint restoration to specific step
- Test export in all formats (JSON, Markdown, HTML)

---

## Sequence Diagram

```mermaid
sequenceDiagram
    participant User
    participant CLI
    participant ReplaySession
    participant Database
    participant FileSystem

    User->>CLI: cf work replay <run-id>
    CLI->>Database: Load execution trace
    Database-->>CLI: Steps, LLM interactions, file ops
    CLI->>ReplaySession: Create session
    
    loop Interactive Navigation
        ReplaySession->>Database: Get step N details
        Database-->>ReplaySession: Step data
        ReplaySession->>FileSystem: Reconstruct file state
        FileSystem-->>ReplaySession: File contents
        ReplaySession->>User: Display step (diff, logs, LLM)
        User->>ReplaySession: Navigate (n/p/j/q)
    end
    
    User->>CLI: cf work diff <run-id>
    CLI->>Database: Load file operations
    Database-->>CLI: All file changes
    CLI->>User: Display unified diff
    
    User->>CLI: cf work export-trace <run-id>
    CLI->>Database: Load complete trace
    Database-->>CLI: Full execution data
    CLI->>FileSystem: Write export file
    CLI->>User: Export complete
```

---

## File References

**Files to modify**:
- file:codeframe/core/workspace.py - Add new tables for execution_steps, llm_interactions, file_operations
- file:codeframe/core/diagnostics.py - Extend RunLogger with step/LLM/file operation logging
- file:codeframe/core/executor.py - Add execution recording hooks
- file:codeframe/core/agent.py - Add step logging for planning and verification
- file:codeframe/cli/app.py - Add replay, diff, export-trace, rerun commands
- file:codeframe/persistence/repositories/checkpoint_repository.py - Link checkpoints to steps

**Files to create**:
- file:codeframe/core/replay.py - Replay session management and state reconstruction
- file:tests/core/test_replay.py - Unit tests for replay functionality
- file:tests/cli/test_replay_commands.py - CLI command tests

**Existing infrastructure to leverage**:
- file:codeframe/core/runtime.py - Run lifecycle management
- file:codeframe/core/diagnostics.py - RunLogger and log storage
- file:codeframe/core/executor.py - FileChange tracking
- file:codeframe/persistence/database.py - Database access layer

---

Execution Information

Branch: main
Commit: e768c1d

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.