Skip to content

Add failing tests for #8: template injection bug#382

Draft
Serhan-Asad wants to merge 1 commit intopromptdriven:mainfrom
Serhan-Asad:fix/issue-8
Draft

Add failing tests for #8: template injection bug#382
Serhan-Asad wants to merge 1 commit intopromptdriven:mainfrom
Serhan-Asad:fix/issue-8

Conversation

@Serhan-Asad
Copy link
Contributor

Summary

Adds failing tests that detect the template injection bug reported in #8.

Test Files

  • Unit test: tests/test_issue_8_template_injection.py
  • E2E test: tests/test_e2e_issue_8_template_injection.py

What This PR Contains

  • Failing unit tests that reproduce the reported bug
  • Failing E2E tests that verify the bug at integration level
  • Tests are verified to fail on current code and will pass once the bug is fixed

Root Cause

The load_prompt_template() function in pdd/load_prompt_template.py doesn't call preprocess() to expand <include> directives. When Step 5's template contains <include>docs/prompting_guide.md</include> and the prompting guide has JSON examples with braces like {"type": "module"}, Python's .format(**context) at line 521 of agentic_change_orchestrator.py interprets those braces as template variables, causing KeyError: '\n "type"'.

Test Coverage

The test suite includes:

  1. Unit Tests (test_issue_8_template_injection.py):

  2. E2E Tests (test_e2e_issue_8_template_injection.py):

    • test_orchestrator_step5_with_include_directive: Primary E2E test simulating the exact user workflow
    • test_orchestrator_full_context_with_include: Tests with full orchestrator context
    • test_multiple_includes_in_template: Regression test for multiple includes

Next Steps

  1. Implement the fix in pdd/load_prompt_template.py to call preprocess()
  2. Verify all unit tests pass
  3. Verify all E2E tests pass
  4. Run full test suite to check for regressions
  5. Mark PR as ready for review

Fixes #8


Generated by PDD agentic bug workflow (Steps 1-10)

This commit adds comprehensive test coverage to reproduce and verify
the template injection bug reported in issue promptdriven#8.

Tests added:
- tests/test_issue_8_template_injection.py: Unit tests verifying that
  load_prompt_template() should preprocess <include> directives
- tests/test_e2e_issue_8_template_injection.py: E2E tests simulating
  the full user workflow at step 5 of pdd change

Root cause:
The load_prompt_template() function doesn't call preprocess() to expand
<include> directives. When Step 5's template includes docs/prompting_guide.md
containing JSON examples with braces, Python's .format(**context) interprets
those braces as template variables, causing KeyError.

All tests correctly fail on the current buggy code and will pass once
load_prompt_template() is fixed to call preprocess().

Related to promptdriven#8

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@Serhan-Asad
Copy link
Contributor Author

Step 10: Draft PR Created ✅

What's Included

This PR adds comprehensive failing tests for the template injection bug:

  • Failing unit tests at tests/test_issue_8_template_injection.py (3 tests)

    • test_load_prompt_template_should_preprocess_includes: Verifies that load_prompt_template() should expand <include> directives
    • test_step5_template_has_unprocessed_include: Integration test reproducing the exact scenario
    • test_load_prompt_template_should_call_preprocess: Root cause verification using mocking
  • Failing E2E tests at tests/test_e2e_issue_8_template_injection.py (3 tests)

    • test_orchestrator_step5_with_include_directive: Primary E2E test simulating the exact user workflow
    • test_orchestrator_full_context_with_include: Tests with full orchestrator context
    • test_multiple_includes_in_template: Regression test for multiple includes

Test Results

All 6 tests correctly fail on the current buggy code, detecting that:

  • load_prompt_template() doesn't import preprocess from pdd.preprocess
  • <include> directives remain unprocessed in templates
  • The function returns raw file content instead of preprocessed templates

Next Steps for Maintainers

  1. Review the failing tests to understand the expected behavior
  2. Implement the fix in pdd/load_prompt_template.py to call preprocess()
  3. Verify all 6 tests pass with your fix
  4. Run full test suite to check for regressions
  5. Mark the PR as ready for review

PDD Fix Command

To auto-fix this bug using PDD, first create a prompt file (if one doesn't already exist), then run:

pdd --force fix --loop --max-attempts 5 --verification-program context/load_prompt_template_example.py <prompt_file> pdd/load_prompt_template.py tests/test_issue_8_template_injection.py

Note: A prompt file for load_prompt_template.py needs to be created first. The fix should:

  1. Import preprocess from pdd.preprocess
  2. Call preprocess() on the template content before returning it
  3. This will expand all <include> directives and escape braces in included content

Impact Summary

  • Root Cause: load_prompt_template() doesn't preprocess <include> directives
  • Bug Location: pdd/load_prompt_template.py
  • Frequency: 100% failure rate at step 5 of pdd change workflow
  • Cost Impact: ~$1.27-$1.57 wasted per failed attempt
  • User Experience: Complete workflow blockage

Investigation complete. Generated by PDD agentic bug workflow (Steps 1-10)

@gltanaka
Copy link
Contributor

Reopening: this PR was automatically closed by a force push to main on Feb 9, not intentionally.

@gltanaka gltanaka reopened this Feb 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generating files from default preprocessed prompt name fails: could not determine language

2 participants