feat(discovery): Improve category coverage detection with AI classification

[frankbria]

Summary

The current category coverage detection in LeadAgent._get_category_coverage() uses simple keyword matching to determine which discovery topics have been covered. This approach has limitations:

Current Implementation

category_keywords = {
    "problem": ["problem", "solve", "issue", "challenge", "pain", "need"],
    "users": ["user", "customer", "audience", "target", "who", "people"],
    "features": ["feature", "function", "capability", "able to", "requirement"],
    "constraints": ["constraint", "limit", "budget", "timeline", "restriction"],
    "tech_stack": ["tech", "stack", "language", "framework", "database", "tool"],
}

Limitations

1. False positives: Answer "I don't have any problem with the current solution" would mark "problem" as covered
2. False negatives: Detailed problem descriptions without keyword matches get missed
3. Context ignorance: Doesn't understand semantic meaning of answers
4. Language sensitivity: Only works with English keywords

Proposed Improvements

Option 1: AI-Powered Classification (Recommended)

Use Claude to analyze each answer and determine which categories it addresses:

def _classify_answer_categories(self, answer: str, question: str) -> List[str]:
    """Use AI to classify which categories an answer addresses."""
    prompt = f"""Analyze this Q&A from a software project discovery session.

Question: {question}
Answer: {answer}

Which of these categories does the answer provide meaningful information about?
- problem: What problem the application solves
- users: Who the target users are
- features: What features are required
- constraints: Technical or business constraints
- tech_stack: Preferred technologies

Return only the category names that are meaningfully addressed, comma-separated.
If none apply, return "none".
"""
    # Call AI and parse response

Benefits:

• Semantic understanding of answers
• Handles varied phrasings
• Can detect partial coverage
• Language agnostic

Drawbacks:

• Additional API calls (cost/latency)
• Non-deterministic

Option 2: Enhanced Keyword Matching

Improve keyword lists and add:

• Phrase matching ("target audience", "end users")
• Negative keyword exclusion ("don't have a problem")
• Synonym expansion
• Answer length thresholds per category

Option 3: Hybrid Approach

Use keyword matching first, then AI classification only for ambiguous cases.

Acceptance Criteria

• Category detection correctly identifies topic coverage from semantic meaning
• False positive rate reduced (e.g., "no problem" doesn't mark problem covered)
• Tests updated to verify improved accuracy
• Document the classification approach in docs/discovery-socratic-methodology.md

Related

• PR feat(discovery): implement AI-powered Socratic questioning system #257: Initial Socratic discovery implementation
• Current location: codeframe/agents/lead_agent.py:_get_category_coverage()

[traycerai]

Plan

Observations

The codebase currently uses a structured discovery framework with 5 categories (problem, users, features, constraints, tech_stack) and 10 questions. Discovery completion is determined by checking if all required questions have valid answers (length > 5 chars, not "N/A"). However, there's no semantic analysis to verify if answers actually address their intended categories. The system already integrates Claude via AnthropicProvider for generating contextual questions and PRDs, making AI-powered classification a natural extension.

Approach

Implement AI-powered category coverage detection using Claude to semantically analyze each answer and determine which categories it meaningfully addresses. This will be integrated into the discovery flow to provide real-time feedback on coverage gaps and improve completion detection. The implementation follows the existing patterns in file:codeframe/agents/lead_agent.py and leverages the established AnthropicProvider for consistency.

Implementation Steps

1. Add Category Coverage Detection Method to LeadAgent

Location: file:codeframe/agents/lead_agent.py

Add a new private method _classify_answer_categories() that uses Claude to analyze Q&A pairs:

• Accept parameters: answer: str, question: str, question_category: str
• Build a focused prompt asking Claude to identify which of the 5 categories (problem, users, features, constraints, tech_stack) are meaningfully addressed
• Use self.provider.send_message() following the existing pattern in start_discovery() and generate_prd()
• Parse Claude's response to extract category names
• Return List[str] of covered categories
• Handle edge cases: empty answers, API failures (return empty list or single category based on question_category)
• Log token usage for monitoring

Prompt Design Considerations:

• Provide clear definitions of each category
• Ask for semantic understanding, not keyword matching
• Request comma-separated category names for easy parsing
• Include examples of what constitutes "meaningful" coverage
• Handle negative responses (e.g., "I don't have a problem" should NOT mark problem as covered)

2. Add Category Coverage Tracking Method

Location: file:codeframe/agents/lead_agent.py

Add a new private method _get_category_coverage() that aggregates coverage across all answers:

• Iterate through self._discovery_answers dictionary
• For each question ID, retrieve the corresponding question text from self.discovery_framework.generate_questions()
• Call _classify_answer_categories() for each Q&A pair
• Aggregate results into a dictionary: Dict[str, bool] mapping category names to coverage status
• A category is "covered" if at least one answer meaningfully addresses it
• Return the coverage dictionary

Data Structure:

{
    "problem": True,
    "users": False,
    "features": True,
    "constraints": False,
    "tech_stack": True
}

3. Integrate Coverage Detection into Discovery Status

Location: file:codeframe/agents/lead_agent.py in get_discovery_status() method

Enhance the status dictionary returned by get_discovery_status():

• Add a new field "category_coverage" containing the result from _get_category_coverage()
• Include this field when state is "discovering" or "completed"
• Calculate a new metric "coverage_percentage": (covered_categories / 5) * 100
• Add "uncovered_categories": list of category names not yet covered

Updated Status Structure:

{
    "state": "discovering",
    "answered_count": 3,
    "progress_percentage": 60.0,
    "category_coverage": {
        "problem": True,
        "users": True,
        "features": False,
        "constraints": False,
        "tech_stack": False
    },
    "coverage_percentage": 40.0,
    "uncovered_categories": ["features", "constraints", "tech_stack"],
    ...
}

4. Optional: Enhance Completion Detection

Location: file:codeframe/agents/lead_agent.py in process_discovery_answer() method

Consider enhancing the completion check to require both:

• All required questions answered (existing check via is_discovery_complete())
• All categories meaningfully covered (new check via _get_category_coverage())

Implementation Note: This is optional and should be configurable. Some projects may not need all categories (e.g., no tech_stack preference). Consider adding a configuration flag or making this a soft warning rather than a hard requirement.

5. Add Unit Tests for Category Classification

Location: file:tests/agents/test_lead_agent.py or new file file:tests/agents/test_lead_agent_category_coverage.py

Create comprehensive tests covering:

Test Cases for _classify_answer_categories():

• Test with answer clearly addressing single category
• Test with answer addressing multiple categories
• Test with negative/dismissive answer (e.g., "I don't have a problem")
• Test with vague/short answer
• Test with answer addressing wrong category
• Test API failure handling
• Test token usage logging

Test Cases for _get_category_coverage():

• Test with no answers (all categories uncovered)
• Test with partial coverage (some categories covered)
• Test with full coverage (all categories covered)
• Test with overlapping coverage (multiple answers covering same category)

Test Cases for Integration:

• Test get_discovery_status() includes category_coverage field
• Test coverage_percentage calculation
• Test uncovered_categories list accuracy

Mocking Strategy:

• Mock AnthropicProvider.send_message() to return controlled responses
• Use parametrized tests for different answer scenarios
• Follow existing test patterns in file:tests/discovery/test_discovery_integration.py

6. Add Integration Tests

Location: file:tests/discovery/test_discovery_integration.py

Add new test class TestDiscoveryCategoryCoverage:

• Test category coverage updates as answers are provided
• Test false positive prevention (negative answers don't mark categories as covered)
• Test false negative prevention (detailed answers without keywords are recognized)
• Test coverage across full discovery flow
• Test persistence of coverage data across agent restarts

7. Update Documentation

Location: Create new file file:docs/discovery-socratic-methodology.md

Document the category coverage detection approach:

Sections to Include:

• Overview of the 5 discovery categories
• How AI classification works
• Prompt design and rationale
• Coverage calculation methodology
• Examples of covered vs. uncovered answers
• API cost considerations (additional Claude calls)
• Configuration options (if implemented)
• Troubleshooting common issues

Additional Documentation Updates:

• Update file:sprints/sprint-02-socratic-discovery.md to reference the new feature
• Add entry to file:CHANGELOG.md describing the enhancement
• Update file:CODEFRAME_USER_DOCUMENTATION.md if it covers discovery features

8. Performance and Cost Considerations

Optimization Strategies:

• Lazy Evaluation: Only call _get_category_coverage() when get_discovery_status() is invoked, not after every answer
• Caching: Cache classification results per answer to avoid re-analyzing on subsequent status checks
  • Store in self._discovery_answers as tuples: (answer_text, covered_categories)
  • Invalidate cache only when answer changes
• Batch Processing: Consider batching multiple Q&A pairs into a single Claude API call to reduce latency and cost
• Fallback to Keywords: If API fails, fall back to simple keyword matching as a degraded mode

Cost Analysis:

• Each classification call: ~100-200 input tokens, ~50 output tokens
• For 10 questions: 1,500 tokens total ($0.01 per discovery session)
• Acceptable cost for improved accuracy

9. Error Handling and Logging

Error Scenarios to Handle:

• API authentication failures: Log error, return empty coverage or fall back to question category
• Rate limiting: Implement exponential backoff or return cached/default values
• Malformed responses: Parse defensively, log warnings, return partial results
• Network timeouts: Set reasonable timeout (5-10 seconds), fall back gracefully

Logging Strategy:

• Log each classification call with question ID and detected categories
• Log token usage for monitoring costs
• Log failures with full context for debugging
• Use appropriate log levels: DEBUG for normal flow, WARNING for fallbacks, ERROR for failures

10. Configuration and Feature Flags

Optional Configuration:

Add to file:codeframe/core/config.py:

• ENABLE_AI_CATEGORY_COVERAGE: Boolean flag to enable/disable AI classification
• CATEGORY_COVERAGE_REQUIRED: Boolean flag to require full coverage for completion
• CATEGORY_COVERAGE_CACHE_TTL: Cache duration for classification results

This allows gradual rollout and A/B testing of the feature.

Sequence Diagram

sequenceDiagram
    participant User
    participant LeadAgent
    participant DiscoveryFramework
    participant AnthropicProvider
    participant Database

    User->>LeadAgent: process_discovery_answer(answer)
    LeadAgent->>Database: Save answer
    LeadAgent->>DiscoveryFramework: get_next_question()
    DiscoveryFramework-->>LeadAgent: next_question
    
    User->>LeadAgent: get_discovery_status()
    LeadAgent->>LeadAgent: _get_category_coverage()
    
    loop For each answered question
        LeadAgent->>DiscoveryFramework: Get question text
        LeadAgent->>LeadAgent: _classify_answer_categories(answer, question)
        LeadAgent->>AnthropicProvider: send_message(classification_prompt)
        AnthropicProvider-->>LeadAgent: covered_categories
        LeadAgent->>LeadAgent: Aggregate coverage
    end
    
    LeadAgent-->>User: status with category_coverage

Loading

Implementation Priority

Priority	Component	Rationale
P0	Steps 1-2	Core classification logic
P0	Step 3	Integration into existing flow
P0	Step 5-6	Tests for reliability
P1	Step 7	Documentation for maintainability
P2	Step 4	Optional enhancement to completion logic
P2	Step 8-10	Optimizations and configuration

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase currently uses a structured discovery framework with 5 categories (problem, users, features, constraints, tech_stack) and 10 questions. Discovery completion is determined by checking if all required questions have valid answers (length > 5 chars, not "N/A"). However, there's no semantic analysis to verify if answers actually address their intended categories. The system already integrates Claude via `AnthropicProvider` for generating contextual questions and PRDs, making AI-powered classification a natural extension.

## Approach

Implement AI-powered category coverage detection using Claude to semantically analyze each answer and determine which categories it meaningfully addresses. This will be integrated into the discovery flow to provide real-time feedback on coverage gaps and improve completion detection. The implementation follows the existing patterns in `file:codeframe/agents/lead_agent.py` and leverages the established `AnthropicProvider` for consistency.

## Implementation Steps

### 1. Add Category Coverage Detection Method to LeadAgent

**Location**: `file:codeframe/agents/lead_agent.py`

Add a new private method `_classify_answer_categories()` that uses Claude to analyze Q&A pairs:

- Accept parameters: `answer: str`, `question: str`, `question_category: str`
- Build a focused prompt asking Claude to identify which of the 5 categories (problem, users, features, constraints, tech_stack) are meaningfully addressed
- Use `self.provider.send_message()` following the existing pattern in `start_discovery()` and `generate_prd()`
- Parse Claude's response to extract category names
- Return `List[str]` of covered categories
- Handle edge cases: empty answers, API failures (return empty list or single category based on question_category)
- Log token usage for monitoring

**Prompt Design Considerations**:
- Provide clear definitions of each category
- Ask for semantic understanding, not keyword matching
- Request comma-separated category names for easy parsing
- Include examples of what constitutes "meaningful" coverage
- Handle negative responses (e.g., "I don't have a problem" should NOT mark problem as covered)

### 2. Add Category Coverage Tracking Method

**Location**: `file:codeframe/agents/lead_agent.py`

Add a new private method `_get_category_coverage()` that aggregates coverage across all answers:

- Iterate through `self._discovery_answers` dictionary
- For each question ID, retrieve the corresponding question text from `self.discovery_framework.generate_questions()`
- Call `_classify_answer_categories()` for each Q&A pair
- Aggregate results into a dictionary: `Dict[str, bool]` mapping category names to coverage status
- A category is "covered" if at least one answer meaningfully addresses it
- Return the coverage dictionary

**Data Structure**:
```python
{
    "problem": True,
    "users": False,
    "features": True,
    "constraints": False,
    "tech_stack": True
}
```

### 3. Integrate Coverage Detection into Discovery Status

**Location**: `file:codeframe/agents/lead_agent.py` in `get_discovery_status()` method

Enhance the status dictionary returned by `get_discovery_status()`:

- Add a new field `"category_coverage"` containing the result from `_get_category_coverage()`
- Include this field when `state` is "discovering" or "completed"
- Calculate a new metric `"coverage_percentage"`: (covered_categories / 5) * 100
- Add `"uncovered_categories"`: list of category names not yet covered

**Updated Status Structure**:
```python
{
    "state": "discovering",
    "answered_count": 3,
    "progress_percentage": 60.0,
    "category_coverage": {
        "problem": True,
        "users": True,
        "features": False,
        "constraints": False,
        "tech_stack": False
    },
    "coverage_percentage": 40.0,
    "uncovered_categories": ["features", "constraints", "tech_stack"],
    ...
}
```

### 4. Optional: Enhance Completion Detection

**Location**: `file:codeframe/agents/lead_agent.py` in `process_discovery_answer()` method

Consider enhancing the completion check to require both:
- All required questions answered (existing check via `is_discovery_complete()`)
- All categories meaningfully covered (new check via `_get_category_coverage()`)

**Implementation Note**: This is optional and should be configurable. Some projects may not need all categories (e.g., no tech_stack preference). Consider adding a configuration flag or making this a soft warning rather than a hard requirement.

### 5. Add Unit Tests for Category Classification

**Location**: `file:tests/agents/test_lead_agent.py` or new file `file:tests/agents/test_lead_agent_category_coverage.py`

Create comprehensive tests covering:

**Test Cases for `_classify_answer_categories()`**:
- Test with answer clearly addressing single category
- Test with answer addressing multiple categories
- Test with negative/dismissive answer (e.g., "I don't have a problem")
- Test with vague/short answer
- Test with answer addressing wrong category
- Test API failure handling
- Test token usage logging

**Test Cases for `_get_category_coverage()`**:
- Test with no answers (all categories uncovered)
- Test with partial coverage (some categories covered)
- Test with full coverage (all categories covered)
- Test with overlapping coverage (multiple answers covering same category)

**Test Cases for Integration**:
- Test `get_discovery_status()` includes category_coverage field
- Test coverage_percentage calculation
- Test uncovered_categories list accuracy

**Mocking Strategy**:
- Mock `AnthropicProvider.send_message()` to return controlled responses
- Use parametrized tests for different answer scenarios
- Follow existing test patterns in `file:tests/discovery/test_discovery_integration.py`

### 6. Add Integration Tests

**Location**: `file:tests/discovery/test_discovery_integration.py`

Add new test class `TestDiscoveryCategoryCoverage`:

- Test category coverage updates as answers are provided
- Test false positive prevention (negative answers don't mark categories as covered)
- Test false negative prevention (detailed answers without keywords are recognized)
- Test coverage across full discovery flow
- Test persistence of coverage data across agent restarts

### 7. Update Documentation

**Location**: Create new file `file:docs/discovery-socratic-methodology.md`

Document the category coverage detection approach:

**Sections to Include**:
- Overview of the 5 discovery categories
- How AI classification works
- Prompt design and rationale
- Coverage calculation methodology
- Examples of covered vs. uncovered answers
- API cost considerations (additional Claude calls)
- Configuration options (if implemented)
- Troubleshooting common issues

**Additional Documentation Updates**:
- Update `file:sprints/sprint-02-socratic-discovery.md` to reference the new feature
- Add entry to `file:CHANGELOG.md` describing the enhancement
- Update `file:CODEFRAME_USER_DOCUMENTATION.md` if it covers discovery features

### 8. Performance and Cost Considerations

**Optimization Strategies**:

- **Lazy Evaluation**: Only call `_get_category_coverage()` when `get_discovery_status()` is invoked, not after every answer
- **Caching**: Cache classification results per answer to avoid re-analyzing on subsequent status checks
  - Store in `self._discovery_answers` as tuples: `(answer_text, covered_categories)`
  - Invalidate cache only when answer changes
- **Batch Processing**: Consider batching multiple Q&A pairs into a single Claude API call to reduce latency and cost
- **Fallback to Keywords**: If API fails, fall back to simple keyword matching as a degraded mode

**Cost Analysis**:
- Each classification call: ~100-200 input tokens, ~50 output tokens
- For 10 questions: ~1,500 tokens total (~$0.01 per discovery session)
- Acceptable cost for improved accuracy

### 9. Error Handling and Logging

**Error Scenarios to Handle**:

- API authentication failures: Log error, return empty coverage or fall back to question category
- Rate limiting: Implement exponential backoff or return cached/default values
- Malformed responses: Parse defensively, log warnings, return partial results
- Network timeouts: Set reasonable timeout (5-10 seconds), fall back gracefully

**Logging Strategy**:
- Log each classification call with question ID and detected categories
- Log token usage for monitoring costs
- Log failures with full context for debugging
- Use appropriate log levels: DEBUG for normal flow, WARNING for fallbacks, ERROR for failures

### 10. Configuration and Feature Flags

**Optional Configuration**:

Add to `file:codeframe/core/config.py`:
- `ENABLE_AI_CATEGORY_COVERAGE`: Boolean flag to enable/disable AI classification
- `CATEGORY_COVERAGE_REQUIRED`: Boolean flag to require full coverage for completion
- `CATEGORY_COVERAGE_CACHE_TTL`: Cache duration for classification results

This allows gradual rollout and A/B testing of the feature.

## Sequence Diagram

```mermaid
sequenceDiagram
    participant User
    participant LeadAgent
    participant DiscoveryFramework
    participant AnthropicProvider
    participant Database

    User->>LeadAgent: process_discovery_answer(answer)
    LeadAgent->>Database: Save answer
    LeadAgent->>DiscoveryFramework: get_next_question()
    DiscoveryFramework-->>LeadAgent: next_question
    
    User->>LeadAgent: get_discovery_status()
    LeadAgent->>LeadAgent: _get_category_coverage()
    
    loop For each answered question
        LeadAgent->>DiscoveryFramework: Get question text
        LeadAgent->>LeadAgent: _classify_answer_categories(answer, question)
        LeadAgent->>AnthropicProvider: send_message(classification_prompt)
        AnthropicProvider-->>LeadAgent: covered_categories
        LeadAgent->>LeadAgent: Aggregate coverage
    end
    
    LeadAgent-->>User: status with category_coverage
```

## Implementation Priority

| Priority | Component | Rationale |
|----------|-----------|-----------|
| P0 | Steps 1-2 | Core classification logic |
| P0 | Step 3 | Integration into existing flow |
| P0 | Step 5-6 | Tests for reliability |
| P1 | Step 7 | Documentation for maintainability |
| P2 | Step 4 | Optional enhancement to completion logic |
| P2 | Step 8-10 | Optimizations and configuration |

---

Execution Information

Branch: main
Commit: 64973be

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Related PRs

#193 - Fix discovery questions repeating in staging [merged]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!