Add failing tests for #486: --quiet flag not suppressing output by Serhan-Asad · Pull Request #489 · promptdriven/pdd

Serhan-Asad · 2026-02-11T17:57:28Z

Summary

Adds failing tests that detect the bug reported in #486.

Test Files

Unit tests: tests/test_quiet_mode.py
E2E test: tests/test_e2e_issue_486_quiet_flag.py

What This PR Contains

5 failing unit tests covering preprocess(), load_prompt_template(), and llm_invoke quiet mode
1 failing E2E test exercising the full pdd --quiet generate CLI path
2 passing regression guard tests ensuring normal (non-quiet) output is preserved
All bug-detecting tests are verified to fail on current code and will pass once the bug is fixed

Root Cause

The --quiet flag is defined in the CLI but never propagated to preprocess.py, load_prompt_template.py, or llm_invoke.py. These modules unconditionally emit Rich panels, INFO logs, and status messages.

Next Steps

Implement the fix in preprocess.py, load_prompt_template.py, and llm_invoke.py
Verify all unit tests pass
Verify the E2E test passes
Run full test suite
Mark PR as ready for review

Fixes #486

Generated by PDD agentic bug workflow

…en#486) Unit tests verify that preprocess(), load_prompt_template(), and llm_invoke respect the quiet flag. E2E test confirms the full CLI path still emits Rich panels and INFO logs despite --quiet. All bug-detecting tests fail on current code, confirming the bug. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fixes promptdriven#486

Monkey-patch rich.console.Console.print, click.echo, and click.secho to suppress all non-error output when --quiet is passed. Only errors pass through; confirmations still show for safety. Also suppresses Python logging (CRITICAL level) and LiteLLM verbose output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

auto_update() makes a real network call during CliRunner.invoke(), causing 60s+ timeouts in CI. Mocking it brings test time to ~2s. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The generate command calls CloudConfig.get_jwt_token() before requests.post, triggering interactive device_flow auth in CI. Mocking it prevents the 60s timeout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Adds test coverage (unit + E2E) for issue #486 where --quiet does not suppress noisy output, and introduces initial plumbing to propagate/enable quiet behavior across CLI, preprocessing, template loading, and LLM invocation.

Changes:

Added new unit tests and an E2E test asserting --quiet suppresses non-essential output while still surfacing errors.
Implemented quiet-mode propagation via quiet params and PDD_QUIET env var, plus global monkey-patching for Rich/Click output.
Adjusted logging verbosity (several info/warning → debug) and changed error-handling expectations to always show errors.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tests/test_quiet_mode.py	New unit tests for quiet behavior in preprocess/template loading/logging and CLI generate path
tests/test_e2e_issue_486_quiet_flag.py	New E2E CLI test validating `pdd --quiet generate` suppresses known noisy strings
tests/test_core_errors.py	Updates expectations so errors are still displayed in quiet mode
tests/test_code_generator_main.py	Updates preprocess mock signature to accept `quiet`
tests/conftest.py	Adds autouse fixture attempting to reset quiet/logging global state between tests
pdd/quiet.py	Introduces global quiet-mode monkey-patching for Rich/Click output
pdd/preprocess.py	Adds `quiet` param + env-based quiet detection to suppress panels/messages
pdd/load_prompt_template.py	Adds `quiet` param + env-based quiet detection to suppress success/not-found messages
pdd/llm_invoke.py	Adds `set_quiet_mode()` and reduces some log noise; suppresses fallback messages in quiet
pdd/core/errors.py	Changes behavior to always show errors even when quiet
pdd/core/cli.py	Enables quiet mode by setting env var, calling `set_quiet_mode`, and enabling global output patching
pdd/code_generator_main.py	Propagates `quiet` into preprocess calls and suppresses some fallback notices

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-11T20:45:22Z

tests/conftest.py

+    if old_last_resort is not None:
+        logging.lastResort = old_last_resort
+    elif logging.lastResort is None:
+        logging.lastResort = logging.StreamHandler()


If old_last_resort was None, the fixture should restore it to None, not create a new StreamHandler(). Creating a new handler changes global logging behavior and can mask/introduce logging output differences across tests. Consider logging.lastResort = old_last_resort unconditionally (or explicitly set back to None when it was None).

Suggested change

if old_last_resort is not None:

logging.lastResort = old_last_resort

elif logging.lastResort is None:

logging.lastResort = logging.StreamHandler()

logging.lastResort = old_last_resort

Copilot · 2026-02-11T20:45:23Z

pdd/core/errors.py

+    elif isinstance(exception, click.UsageError):
+         console.print(f"  [error]Usage Error:[/error] {exception}", style="error")
+         raise exception


The click.UsageError branch appears to be mis-indented relative to the surrounding elif chain (extra leading whitespace on lines 56–57). In Python, inconsistent indentation here can raise an IndentationError or change control flow. Align the block to standard 4-space indentation (8 spaces inside the elif).

Copilot · 2026-02-11T20:45:23Z

pdd/quiet.py

+        if not _quiet or (args and _is_error(args[0])):
+            _original_console_print(self, *args, **kwargs)
+


The Rich Console.print suppression only checks args[0] for error-ness. If an error indicator is in a later positional arg (or the printable is a Rich renderable like Panel containing an error), it will be suppressed even though it’s an error. Consider checking all args (e.g., any arg matches _is_error(str(arg))) and/or allowing through when kwargs indicate error styling.

Suggested change

if not _quiet or (args and _is_error(args[0])):

_original_console_print(self, *args, **kwargs)

# Allow output if quiet mode is off

if not _quiet:

_original_console_print(self, *args, **kwargs)

return

# In quiet mode, allow through if any positional arg looks like an error

if any(_is_error(arg) for arg in args):

_original_console_print(self, *args, **kwargs)

return

# Also allow through if kwargs indicate error styling (e.g., red text)

style = kwargs.get("style")

if isinstance(style, str) and "red" in style:

_original_console_print(self, *args, **kwargs)

return

# Otherwise, suppress output in quiet mode

return

Copilot · 2026-02-11T20:45:23Z

pdd/llm_invoke.py

+def set_quiet_mode():
+    """Suppress all log output, LiteLLM verbose output, and warnings in quiet mode."""
+    # Set all loggers to CRITICAL to suppress everything below
+    logger.setLevel(logging.CRITICAL)
+    litellm_logger.setLevel(logging.CRITICAL)
+    # Also raise handler levels so propagation doesn't leak through
+    for handler in logger.handlers:
+        handler.setLevel(logging.CRITICAL)
+    for handler in litellm_logger.handlers:
+        handler.setLevel(logging.CRITICAL)
+    # Suppress LiteLLM's own print-based verbose output
+    litellm.set_verbose = False
+    # Suppress the root logger and all its handlers
+    root = logging.getLogger()
+    root.setLevel(logging.CRITICAL)
+    for handler in root.handlers:
+        handler.setLevel(logging.CRITICAL)
+    # Disable the lastResort handler that prints WARNING: lines
+    logging.lastResort = None
+    # Suppress Python warnings (Pydantic serialization warnings, etc.)
+    import warnings
+    warnings.filterwarnings("ignore")


set_quiet_mode() makes several process-global changes (root logger level/handlers, logging.lastResort, and warnings.filterwarnings('ignore')). This is hard to safely undo in test runs and can cause unrelated tests to behave differently. Consider implementing a reversible quiet-mode (store/restore previous logger levels, lastResort, and warnings filters) rather than permanently mutating global state.

Copilot · 2026-02-11T20:45:24Z

pdd/code_generator_main.py

                p_output.write_text(final_content, encoding="utf-8")
-                if verbose or not quiet:
-                    console.print(f"Generated code saved to: [green]{p_output.resolve()}[/green]")
+                click.echo(f"Generated code saved to: {p_output.resolve()}")


This used to be guarded by if verbose or not quiet: but now always prints. In non-CLI/programmatic usages where quiet=True is passed but global Click/Rich monkey-patching isn’t enabled, this will incorrectly emit output in quiet mode. Consider restoring a local quiet guard around this message (or route through a project-wide output helper that respects quiet).

Suggested change

click.echo(f"Generated code saved to: {p_output.resolve()}")

if verbose or not quiet:

click.echo(f"Generated code saved to: {p_output.resolve()}")

Copilot · 2026-02-11T20:45:24Z

tests/test_quiet_mode.py

+"""
+
+import logging
+from unittest.mock import patch, MagicMock, call


MagicMock and call are imported but not used anywhere in this new test module. Removing unused imports will reduce lint noise and keep the test file focused.

Suggested change

from unittest.mock import patch, MagicMock, call

from unittest.mock import patch

Copilot · 2026-02-11T20:45:24Z

tests/test_e2e_issue_486_quiet_flag.py

+"""
+
+import pytest
+from unittest.mock import patch, MagicMock


MagicMock is imported but not used in this test file. Removing it will keep the imports clean and avoid lint warnings.

Suggested change

from unittest.mock import patch, MagicMock

from unittest.mock import patch

Copilot · 2026-02-11T20:45:25Z

tests/test_quiet_mode.py

+    """Tests that llm_invoke logger level is raised to WARNING when quiet=True."""
+
+    def test_set_quiet_mode_raises_log_level(self):
+        """set_quiet_mode() should set both loggers to WARNING level."""


The docstrings say set_quiet_mode() raises logger level to WARNING, but the implementation sets loggers to CRITICAL. Updating the docstrings to match the actual behavior will prevent confusion when maintaining these tests.

Suggested change

"""Tests that llm_invoke logger level is raised to WARNING when quiet=True."""

def test_set_quiet_mode_raises_log_level(self):

"""set_quiet_mode() should set both loggers to WARNING level."""

"""Tests that llm_invoke logger level is raised to CRITICAL when quiet=True."""

def test_set_quiet_mode_raises_log_level(self):

"""set_quiet_mode() should set both loggers to CRITICAL level."""

- Fix lastResort restore in conftest.py (was creating new StreamHandler instead of restoring None) - Fix indentation on UsageError branch in errors.py - Restore quiet guard on "Generated code saved to" message in code_generator_main.py - Remove unused imports (MagicMock, call) from test files - Fix docstrings: say CRITICAL not WARNING to match actual log level Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Serhan-Asad and others added 6 commits February 10, 2026 21:25

fix: --quiet flag does not suppress output

ee846e0

Fixes promptdriven#486

fix: mock auto_update in E2E quiet flag tests to prevent timeout

51d73a6

auto_update() makes a real network call during CliRunner.invoke(), causing 60s+ timeouts in CI. Mocking it brings test time to ~2s. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ci: trigger re-run on latest commit

eb43249

fix: mock CloudConfig.get_jwt_token to prevent CI timeout

58e64f5

The generate command calls CloudConfig.get_jwt_token() before requests.post, triggering interactive device_flow auth in CI. Mocking it prevents the 60s timeout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

gltanaka requested a review from Copilot February 11, 2026 20:42

Copilot AI reviewed Feb 11, 2026

View reviewed changes

Serhan-Asad marked this pull request as draft February 11, 2026 22:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add failing tests for #486: --quiet flag not suppressing output#489

Add failing tests for #486: --quiet flag not suppressing output#489
Serhan-Asad wants to merge 7 commits intopromptdriven:mainfrom
Serhan-Asad:fix/issue-486

Serhan-Asad commented Feb 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		if not _quiet or (args and _is_error(args[0])):
		_original_console_print(self, args, *kwargs)

-        if not _quiet or (args and _is_error(args[0])):
-            _original_console_print(self, *args, **kwargs)
+        # Allow output if quiet mode is off
+        if not _quiet:
+            _original_console_print(self, *args, **kwargs)
+            return
+        # In quiet mode, allow through if any positional arg looks like an error
+        if any(_is_error(arg) for arg in args):
+            _original_console_print(self, *args, **kwargs)
+            return
+        # Also allow through if kwargs indicate error styling (e.g., red text)
+        style = kwargs.get("style")
+        if isinstance(style, str) and "red" in style:
+            _original_console_print(self, *args, **kwargs)
+            return
+        # Otherwise, suppress output in quiet mode
+        return

	click.echo(f"Generated code saved to: {p_output.resolve()}")
	if verbose or not quiet:
	click.echo(f"Generated code saved to: {p_output.resolve()}")

	from unittest.mock import patch, MagicMock, call
	from unittest.mock import patch

Conversation

Serhan-Asad commented Feb 11, 2026

Summary

Test Files

What This PR Contains

Root Cause

Next Steps

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant