Generated tests use fragile try/except ImportError pattern for mocking

[jiaminc-cmu]

Summary

PDD generates tests that use try/except ImportError to conditionally install mocks in sys.modules. This pattern is fragile and breaks when modules exist but require external services (Secret Manager, databases, APIs).

Problem

PDD generates code like this:

# 2. Mock src.config
try:
    import src.config        # If this succeeds, mock is SKIPPED
except ImportError:          
    mock_config = MagicMock()
    sys.modules["src.config"] = mock_config

# 3. Mock src.clients.secrets_client
try:
    import src.clients.secrets_client  # If this succeeds, mock is SKIPPED
except ImportError:                     
    mock_secrets_mod = MagicMock()
    sys.modules["src.clients.secrets_client"] = mock_secrets_mod

The assumption is wrong: This pattern assumes that if a module imports successfully, it doesn't need mocking. But modules can import fine while still requiring external services at runtime.

Real-World Failure

In a generated github_app_2 project:

1. src.clients.secrets_client imports successfully (it's valid Python code)
2. The try/except ImportError block is skipped - no mock installed
3. Test calls get_installation_token() which calls secrets_client.get_github_private_key()
4. The real secrets_client tries to connect to Google Secret Manager
5. Test hangs indefinitely waiting for GCP credentials that don't exist

# This test hangs forever:
tests/test_github_client.py::test_get_installation_token_success

Why It Worked During Sync

During pdd sync, the environment was different:

• Possibly different import order
• Possibly missing dependencies that caused ImportError
• The _run.json shows "tests_passed": 16, "tests_failed": 0

But running the same tests now (with all dependencies installed) causes hangs because the mocks are never applied.

The Anti-Pattern

# BAD: Only mocks if import fails
try:
    import some_module
except ImportError:
    sys.modules["some_module"] = MagicMock()

# GOOD: Always mock external dependencies in tests
@pytest.fixture
def mock_secrets_client():
    with patch("src.clients.github_client.secrets_client") as mock:
        mock.get_github_private_key.return_value = "fake-key"
        yield mock

Impact

1. Tests hang: Real services are called instead of mocks
2. False confidence: Sync reports success but tests don't actually run correctly
3. Environment-dependent: Tests pass in some environments, hang/fail in others
4. Hard to debug: Hangs give no error message, just timeout

Suggested Fix

PDD should:

1. Never use try/except ImportError for mocking - this pattern is fundamentally flawed for testing
2. Always mock external dependencies unconditionally using unittest.mock.patch or pytest fixtures
3. Mock at the point of use, not at import time:

   # Mock where the module is used, not where it's defined
   with patch("src.clients.github_client.secrets_client") as mock:
       ...

4. Identify external dependencies (secrets, databases, APIs, network) and always mock them regardless of import success

Related

This is related to #412 (incorrect sys.modules paths) but is a separate issue about the conditional mocking pattern itself.

Environment

• PDD version: latest (as of 2026-01-27)
• Python: 3.12
• All dependencies installed (github, google-cloud-secret-manager, etc.)

[prompt-driven-github]

🔗 Connect Session Starting!

Session ID: 8a2644a7-c716-4c68-a700-406fdccf6b4f
Triggered by: @jiaminc-cmu
You can interact with this session from the PDD Cloud web UI.
The session will auto-commit and create a PR on shutdown.

[prompt-driven-github]

🔗 Connect Session Starting!

Session ID: 42bc2e87-c456-4ce9-b2bd-51a4dc137a5e
Triggered by: @jiaminc-cmu
You can interact with this session from the PDD Cloud web UI.
The session will auto-commit and create a PR on shutdown.

[prompt-driven-github]

PDD Connect Session Ready

Your web-based PDD Connect session is now accessible:

Open Session

Session ID: 8a2644a7
This link is temporary and will expire when the session ends (max 4 hours).

Security: Do not share this link. It grants full access to your session.

[prompt-driven-github]

PDD Connect Session Ready

Your web-based PDD Connect session is now accessible:

Open Session

Session ID: 42bc2e87
This link is temporary and will expire when the session ends (max 4 hours).

Security: Do not share this link. It grants full access to your session.

[prompt-driven-github]

Connect Session Stopping

The connect label was removed. Sent shutdown signal to 2 active session(s). Sessions will auto-commit, push changes, and create a PR before exiting.

[prompt-driven-github]

🚀 Job Queued!

Job ID: 2B1OWLF2JTxNEFdioLII
Triggered by: @jiaminc-cmu
Label: bug

[prompt-driven-github]

Step 1: Duplicate Check

Status: No duplicates found

Search Performed

• Searched for: "mock", "test", "ImportError", "hang", "secrets", "external service", "try/except", "conditional"
• Issues reviewed: 20+ issues examined in detail

Findings

This appears to be a new, distinct issue. While several related test generation issues were found, none are true duplicates:

Issue	Relationship	Why Not Duplicate
#412	Related	Focuses on incorrect sys.modules paths (e.g., "config" vs "src.config"). #413 is about the conditional logic that prevents mocks from being installed at all when imports succeed. Different root causes.
#396	Related umbrella	Broad feature request for general test isolation improvements. #413 is a specific bug about the try/except ImportError anti-pattern causing tests to hang.
#349	Related (closed)	About module-level sys.modules pollution causing collection failures. #413 is about conditional mocking logic causing tests to call real external services and hang. Different symptoms and root causes.
#342	Related (closed)	About sys.path ordering causing imports from site-packages vs local code. Different issue entirely.

Key distinction: Issue #413 uniquely describes:

1. The problematic try/except ImportError pattern that conditionally decides whether to mock
2. Tests hanging indefinitely when real external services (Secret Manager, databases) are called instead of mocks
3. The flawed assumption that "module imports successfully = doesn't need mocking"
4. Environment-dependent behavior (works when dependencies missing, hangs when installed)

This is a distinct code generation anti-pattern not covered by the other issues, which focus on path correctness (#412), general isolation (#396), or cleanup pollution (#349).

---

Proceeding to Step 2: Documentation Check

[prompt-driven-github]

[prompt-driven-github]

Step 2: Documentation Check

Status: Confirmed Bug

Documentation Reviewed

• Main README.md
• context/pytest_isolation_example.py (reference implementation for test generation)
• pdd/prompts/agentic_bug_step7_generate_LLM.prompt (test generation prompt)
• examples/hello/tests/test_hello.py (generated test example)
• Test generation implementation in pdd/generate_test.py

Findings

The reported behavior contradicts the existing documentation and best practices established in the codebase. This is a confirmed bug.

Evidence of Contradictory Documentation

1. The context/pytest_isolation_example.py file explicitly documents CORRECT patterns:

This file serves as a reference implementation for test generation and includes extensive documentation about proper mocking patterns. It specifically addresses sys.modules mocking:

• PATTERN 4 (line 111-129): Shows the CORRECT way to mock sys.modules with proper save/restore:

• PATTERN 7 (lines 189-241): Explicitly documents module-level mocking with save/restore pattern, not try/except ImportError

• PATTERN 9 (lines 330-368): Documents the ANTI-PATTERN of using patcher.start() without stop(), which is conceptually similar to the reported issue

2. Generated tests use the anti-pattern:

The file examples/hello/tests/test_hello.py (lines 9-14) demonstrates the problematic pattern:

This is a milder version of the issue, but shows the same fundamental flaw: conditional behavior based on ImportError.

The Core Problem

The issue describes PDD generating code like:

This pattern has a fundamental flaw:

• Assumption: If a module imports successfully, it doesn't need mocking
• Reality: Modules can import fine while still requiring external services (databases, APIs, Secret Manager)
• Result: Tests call real external services instead of mocks, causing hangs/failures

Why This Is a Bug, Not a Feature Request

1. Contradicts documented best practices: The context/pytest_isolation_example.py file shows the correct patterns
2. Causes test failures: Tests hang or fail when external services are unavailable
3. Environment-dependent behavior: Tests pass in some environments, fail in others
4. Violates test isolation principles: Tests should never depend on external services

The documented approach is to always mock external dependencies unconditionally using fixtures or context managers, not to conditionally mock based on import success.

---

Proceeding to Step 3: Triage