diff --git a/README.md b/README.md index 94836e56..7fdd96bf 100644 --- a/README.md +++ b/README.md @@ -591,11 +591,31 @@ flowchart TB - **[`detect`](#10-detect)**: Analyzes prompts to determine which ones need changes based on a description - **[`conflicts`](#11-conflicts)**: Finds and suggests resolutions for conflicts between two prompt files - **[`trace`](#13-trace)**: Finds the corresponding line number in a prompt file for a given code line +- **[`story-test`](#21-story-test)**: Validates prompt changes against user stories ### Utility Commands - **[`auth`](#18-auth)**: Manages authentication with PDD Cloud - **[`sessions`](#19-pdd-sessions---manage-remote-sessions)**: Manage remote sessions for `connect` +### User Story Prompt Tests +PDD can validate prompt changes against user stories stored as Markdown files. This uses `detect` under the hood: a story **passes** when `detect` returns no required prompt changes. + +Defaults: +- Stories live in `user_stories/` and match `story__*.md`. +- Prompts are loaded from `prompts/` (excluding `*_llm.prompt` by default). + +Overrides: +- `PDD_USER_STORIES_DIR` sets the stories directory. +- `PDD_PROMPTS_DIR` sets the prompts directory. + +Commands: +- `pdd story-test` runs the validation suite. +- `pdd change` runs story validation after prompt modifications and fails if any story fails. +- `pdd fix user_stories/story__*.md` applies a single story to prompts and re-validates it. + +Template: +- See `user_stories/story__template.md` for a starter format. + ## Global Options These options can be used with any command: @@ -1738,6 +1758,13 @@ pdd [GLOBAL OPTIONS] fix [OPTIONS] pdd [GLOBAL OPTIONS] fix --manual [OPTIONS] PROMPT_FILE CODE_FILE UNIT_TEST_FILE ERROR_FILE ``` +**User Story Fix Mode:** +``` +pdd [GLOBAL OPTIONS] fix user_stories/story__my_story.md +``` + +This mode treats the story file as a change request for prompts. It runs `detect` to identify impacted prompts, applies prompt updates, and re-validates the story. + #### Manual Mode Arguments - `PROMPT_FILE`: The filename of the prompt file that generated the code under test. - `CODE_FILE`: The filename of the code file to be fixed. @@ -1964,6 +1991,8 @@ Options: - `--output LOCATION`: Specify where to save the modified prompt file. The default file name is `modified_.prompt`. If an environment variable `PDD_CHANGE_OUTPUT_PATH` is set, the file will be saved in that path unless overridden by this option. - `--csv`: Use a CSV file for the change prompts instead of a single change prompt file. The CSV file should have columns: `prompt_name` and `change_instructions`. When this option is used, `INPUT_PROMPT_FILE` is not needed, and `INPUT_CODE` should be the directory where the code files are located. The command expects prompt names in the CSV to follow the `_.prompt` convention. For each `prompt_name` in the CSV, it will look for the corresponding code file (e.g., `.`) within the specified `INPUT_CODE` directory. Output files will overwrite existing files unless `--output LOCATION` is specified. If `LOCATION` is a directory, the modified prompt files will be saved inside this directory using the default naming convention otherwise, if a csv filename is specified the modified prompts will be saved in that CSV file with columns 'prompt_name' and 'modified_prompt'. +**User Story Validation:** After prompt modifications (agentic or manual), `pdd change` runs user story tests when story files exist. It fails the command if any story indicates required prompt changes. Stories default to `user_stories/story__*.md` and can be overridden with `PDD_USER_STORIES_DIR`. + Example (manual single prompt change): ``` pdd [GLOBAL OPTIONS] change --manual --output modified_factorial_calculator_python.prompt changes_factorial.prompt src/factorial_calculator.py factorial_calculator_python.prompt @@ -2499,6 +2528,27 @@ pdd firecrawl-cache check # Check if a URL is cached **When to use**: Caching is automatic. Use `stats` to check cache status, `info` to view configuration, `check` to verify if a URL is cached, or `clear` to force re-scraping all URLs. +### 21. story-test + +Validate prompt changes against user stories stored as Markdown files in `user_stories/`. A story **passes** when `detect` finds no required prompt changes. + +**Usage:** +```bash +pdd [GLOBAL OPTIONS] story-test [OPTIONS] +``` + +**Options:** +- `--stories-dir DIR`: Directory containing `story__*.md` files (default: `user_stories/`). +- `--prompts-dir DIR`: Directory containing `.prompt` files (default: `prompts/`). +- `--include-llm`: Include `*_llm.prompt` files in validation. +- `--fail-fast/--no-fail-fast`: Stop on the first failing story (default: `--fail-fast`). + +**Examples:** +```bash +pdd story-test +PDD_USER_STORIES_DIR=stories pdd story-test --prompts-dir prompts +``` + ## Example Review Process When the global `--review-examples` option is used with any command, PDD will present potential few-shot examples that might be used for the current operation. The review process follows these steps: @@ -2658,6 +2708,7 @@ PDD uses several environment variables to customize its behavior: **Note**: When using `.pddrc` configuration, context-specific settings take precedence over these global environment variables. - **`PDD_PROMPTS_DIR`**: Default directory where prompt files are located (default: "prompts"). +- **`PDD_USER_STORIES_DIR`**: Default directory where user story files are located (default: "user_stories"). - **`PDD_GENERATE_OUTPUT_PATH`**: Default path for the `generate` command. - **`PDD_EXAMPLE_OUTPUT_PATH`**: Default path for the `example` command. - **`PDD_TEST_OUTPUT_PATH`**: Default path for the unit test file. diff --git a/pdd/change_main.py b/pdd/change_main.py index 6cbfc744..f4579a7f 100644 --- a/pdd/change_main.py +++ b/pdd/change_main.py @@ -22,6 +22,7 @@ from .change import change as change_func from .process_csv_change import process_csv_change from .get_extension import get_extension +from .user_story_tests import run_user_story_tests, discover_prompt_files # Set up logging logger = logging.getLogger(__name__) @@ -487,7 +488,71 @@ def change_main( logger.error(msg, exc_info=True) return msg, total_cost, model_name or "" - # --- 5. Final User Feedback --- + # --- 5. User Story Validation (Optional) --- + if (use_csv or success) and not ctx.obj.get("skip_user_stories", False): + prompts_dir = resolved_config.get("prompts_dir") or os.environ.get("PDD_PROMPTS_DIR") or "prompts" + stories_dir = os.environ.get("PDD_USER_STORIES_DIR") or "user_stories" + validation_prompt_files = None + validation_prompts_dir = Path(prompts_dir) + output_is_csv = False + + if use_csv and output_path_obj: + output_is_csv = output_path_obj.suffix.lower() == ".csv" + + if output_is_csv: + if not quiet: + rprint("[yellow]Skipping user story validation: output is CSV, no prompt files written.[/yellow]") + passed = True + story_cost = 0.0 + story_model = "" + else: + override_dir = None + if use_csv: + if "output_dir" in locals(): + override_dir = output_dir + elif output_path_obj: + if output_path_obj.is_dir() or (not output_path_obj.exists() and not output_path_obj.suffix): + override_dir = output_path_obj + else: + override_dir = output_path_obj.parent + else: + if output_path_obj: + override_dir = output_path_obj.parent + + if override_dir: + override_prompts = discover_prompt_files(str(override_dir)) + base_prompts = discover_prompt_files(str(validation_prompts_dir)) + merged: List[Path] = [] + seen = set() + for pf in override_prompts + base_prompts: + key = pf.name.lower() + if key in seen: + continue + merged.append(pf) + seen.add(key) + validation_prompt_files = merged + + passed, _, story_cost, story_model = run_user_story_tests( + prompts_dir=str(validation_prompts_dir) if validation_prompt_files is None else None, + prompt_files=validation_prompt_files, + stories_dir=stories_dir, + strength=strength, + temperature=temperature, + time=time_budget, + verbose=not quiet, + quiet=quiet, + fail_fast=True, + ) + total_cost += story_cost + if story_model: + model_name = model_name or story_model + if not passed: + msg = "User story validation failed. Review detect results for details." + if not quiet: + rprint(f"[bold red]Error:[/bold red] {msg}") + return msg, total_cost, model_name or "" + + # --- 6. Final User Feedback --- # Show summary if not quiet AND (it was CSV mode OR non-CSV mode succeeded) if not quiet and (use_csv or success): rprint("[bold green]Prompt modification completed successfully.[/bold green]") diff --git a/pdd/commands/__init__.py b/pdd/commands/__init__.py index a7f7ba64..325afcdc 100644 --- a/pdd/commands/__init__.py +++ b/pdd/commands/__init__.py @@ -7,7 +7,7 @@ from .fix import fix from .modify import split, change, update from .maintenance import sync, auto_deps, setup -from .analysis import detect_change, conflicts, bug, crash, trace +from .analysis import detect_change, conflicts, bug, crash, trace, story_test from .connect import connect from .auth import auth_group from .misc import preprocess @@ -35,6 +35,7 @@ def register_commands(cli: click.Group) -> None: cli.add_command(bug) cli.add_command(crash) cli.add_command(trace) + cli.add_command(story_test) cli.add_command(preprocess) cli.add_command(report_core) cli.add_command(install_completion_cmd, name="install_completion") diff --git a/pdd/commands/analysis.py b/pdd/commands/analysis.py index 33f59ea7..f0e2e974 100644 --- a/pdd/commands/analysis.py +++ b/pdd/commands/analysis.py @@ -13,6 +13,7 @@ from ..agentic_bug import run_agentic_bug from ..crash_main import crash_main from ..trace_main import trace_main +from ..user_story_tests import run_user_story_tests from ..track_cost import track_cost from ..core.errors import handle_error from ..operation_log import log_operation @@ -62,6 +63,65 @@ def detect_change( return None +@click.command("story-test") +@click.option( + "--stories-dir", + type=click.Path(file_okay=False, dir_okay=True), + default=None, + help="Directory containing story__*.md files (default: user_stories).", +) +@click.option( + "--prompts-dir", + type=click.Path(file_okay=False, dir_okay=True), + default=None, + help="Directory containing .prompt files (default: prompts).", +) +@click.option( + "--include-llm", + is_flag=True, + default=False, + help="Include *_llm.prompt files in validation.", +) +@click.option( + "--fail-fast/--no-fail-fast", + default=True, + help="Stop on the first failing story.", +) +@click.pass_context +@track_cost +def story_test( + ctx: click.Context, + stories_dir: Optional[str], + prompts_dir: Optional[str], + include_llm: bool, + fail_fast: bool, +) -> Optional[Tuple[Dict[str, Any], float, str]]: + """Validate prompt changes against user stories.""" + try: + obj = get_context_obj(ctx) + passed, results, total_cost, model_name = run_user_story_tests( + prompts_dir=prompts_dir, + stories_dir=stories_dir, + strength=obj.get("strength", 0.2), + temperature=obj.get("temperature", 0.0), + time=obj.get("time", 0.25), + verbose=obj.get("verbose", False), + quiet=obj.get("quiet", False), + fail_fast=fail_fast, + include_llm_prompts=include_llm, + ) + result = { + "passed": passed, + "results": results, + } + return result, total_cost, model_name + except (click.Abort, click.ClickException): + raise + except Exception as exception: + handle_error(exception, "story-test", get_context_obj(ctx).get("quiet", False)) + return None + + @click.command("conflicts") @click.argument("prompt1", type=click.Path(exists=True, dir_okay=False)) @click.argument("prompt2", type=click.Path(exists=True, dir_okay=False)) @@ -310,4 +370,4 @@ def trace( raise except Exception as exception: handle_error(exception, "trace", get_context_obj(ctx).get("quiet", False)) - return None \ No newline at end of file + return None diff --git a/pdd/commands/fix.py b/pdd/commands/fix.py index 609b0f50..bf1025d1 100644 --- a/pdd/commands/fix.py +++ b/pdd/commands/fix.py @@ -73,6 +73,13 @@ def fix( # Determine mode based on first argument is_url = args[0].startswith("http") or "github.com" in args[0] + + def is_user_story_file(path: str) -> bool: + return ( + path.endswith(".md") + and os.path.basename(path).startswith("story__") + and os.path.exists(path) + ) if is_url and not manual: if len(args) > 1: @@ -107,6 +114,32 @@ def fix( return result_dict, cost, model else: + if not manual and len(args) == 1 and is_user_story_file(args[0]): + from ..user_story_tests import run_user_story_fix + + ctx_obj = ctx.obj or {} + success, message, cost, model, changed_files = run_user_story_fix( + ctx=ctx, + story_file=args[0], + prompts_dir=ctx_obj.get("prompts_dir"), + strength=ctx_obj.get("strength", 0.2), + temperature=ctx_obj.get("temperature", 0.0), + time=ctx_obj.get("time", 0.25), + budget=budget, + verbose=ctx_obj.get("verbose", False), + quiet=ctx_obj.get("quiet", False), + ) + if success: + console.print(f"[bold green]User story fix completed:[/bold green] {message}") + else: + console.print(f"[bold red]User story fix failed:[/bold red] {message}") + result_dict = { + "success": success, + "message": message, + "changed_files": changed_files, + } + return result_dict, cost, model + min_args = 3 if loop else 4 if len(args) < min_args: mode_str = "Loop" if loop else "Non-loop" @@ -188,4 +221,4 @@ def fix( except Exception as e: quiet = ctx.obj.get("quiet", False) if ctx.obj else False handle_error(e, "fix", quiet) - ctx.exit(1) \ No newline at end of file + ctx.exit(1) diff --git a/pdd/user_story_tests.py b/pdd/user_story_tests.py new file mode 100644 index 00000000..cfcc7d8c --- /dev/null +++ b/pdd/user_story_tests.py @@ -0,0 +1,280 @@ +from __future__ import annotations + +import os +from pathlib import Path +from typing import Dict, Iterable, List, Optional, Tuple + +from rich import print as rprint + +from .detect_change import detect_change +from .get_extension import get_extension + + +DEFAULT_STORIES_DIR = "user_stories" +DEFAULT_PROMPTS_DIR = "prompts" +STORY_PREFIX = "story__" +STORY_SUFFIX = ".md" + + +def _resolve_stories_dir(stories_dir: Optional[str] = None) -> Path: + resolved = stories_dir or os.environ.get("PDD_USER_STORIES_DIR") or DEFAULT_STORIES_DIR + return Path(resolved) + + +def _resolve_prompts_dir(prompts_dir: Optional[str] = None) -> Path: + resolved = prompts_dir or os.environ.get("PDD_PROMPTS_DIR") or DEFAULT_PROMPTS_DIR + return Path(resolved) + + +def discover_story_files(stories_dir: Optional[str] = None) -> List[Path]: + base_dir = _resolve_stories_dir(stories_dir) + if not base_dir.exists() or not base_dir.is_dir(): + return [] + return sorted(p for p in base_dir.glob(f"{STORY_PREFIX}*{STORY_SUFFIX}") if p.is_file()) + + +def discover_prompt_files( + prompts_dir: Optional[str] = None, + *, + include_llm: bool = False, +) -> List[Path]: + base_dir = _resolve_prompts_dir(prompts_dir) + if not base_dir.exists() or not base_dir.is_dir(): + return [] + prompts = [p for p in base_dir.rglob("*.prompt") if p.is_file()] + if not include_llm: + prompts = [p for p in prompts if not p.name.lower().endswith("_llm.prompt")] + return sorted(prompts) + + +def _read_story(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _build_prompt_name_map(prompt_files: Iterable[Path]) -> Dict[str, Path]: + name_map: Dict[str, Path] = {} + for pf in prompt_files: + name_map[pf.name] = pf + name_map[str(pf)] = pf + name_map[pf.name.lower()] = pf + name_map[str(pf).lower()] = pf + return name_map + + +def _resolve_prompt_path(prompt_name: str, prompt_files: Iterable[Path]) -> Optional[Path]: + name_map = _build_prompt_name_map(prompt_files) + if prompt_name in name_map: + return name_map[prompt_name] + lower = prompt_name.lower() + if lower in name_map: + return name_map[lower] + # Fallback: match by basename if detect output used a short name + for pf in prompt_files: + if pf.name == prompt_name or pf.name.lower() == lower: + return pf + return None + + +def _prompt_to_code_path(prompt_path: Path, prompts_dir: Path) -> Optional[Path]: + try: + rel_path = prompt_path.relative_to(prompts_dir) + except ValueError: + return None + + stem = rel_path.stem + if "_" not in stem: + return None + + basename_part, language = stem.rsplit("_", 1) + try: + extension = get_extension(language) + except ValueError: + return None + extension = extension.lstrip(".") + + code_dir = prompts_dir.parent / "src" + rel_dir = rel_path.parent + if not extension: + return None + code_name = f"{basename_part}.{extension}" + return (code_dir / rel_dir / code_name).resolve() + + +def run_user_story_tests( + *, + prompts_dir: Optional[str] = None, + stories_dir: Optional[str] = None, + story_files: Optional[List[Path]] = None, + prompt_files: Optional[List[Path]] = None, + strength: float = 0.2, + temperature: float = 0.0, + time: float = 0.25, + verbose: bool = False, + quiet: bool = False, + fail_fast: bool = False, + include_llm_prompts: bool = False, +) -> Tuple[bool, List[Dict[str, object]], float, str]: + """ + Run user story tests by calling detect_change on each story. + + A story passes if detect_change returns an empty changes_list. + """ + prompt_files = prompt_files or discover_prompt_files(prompts_dir, include_llm=include_llm_prompts) + story_files = story_files or discover_story_files(stories_dir) + + if not story_files: + return True, [], 0.0, "" + if not prompt_files: + msg = "No prompt files found to validate user stories." + if not quiet: + rprint(f"[bold yellow]{msg}[/bold yellow]") + return False, [], 0.0, "" + + total_cost = 0.0 + model_name = "" + results: List[Dict[str, object]] = [] + all_passed = True + + for story_path in story_files: + story_content = _read_story(story_path) + changes_list, cost, model = detect_change( + [str(p) for p in prompt_files], + story_content, + strength, + temperature, + time, + verbose=verbose, + ) + total_cost += cost + model_name = model or model_name + passed = len(changes_list) == 0 + if not passed: + all_passed = False + results.append({ + "story": str(story_path), + "passed": passed, + "changes": changes_list, + }) + + if not quiet: + status = "PASS" if passed else "FAIL" + rprint(f"[bold]{status}[/bold] {story_path}") + + if fail_fast and not passed: + break + + return all_passed, results, total_cost, model_name + + +def run_user_story_fix( + *, + ctx: object, + story_file: str, + prompts_dir: Optional[str] = None, + strength: float = 0.2, + temperature: float = 0.0, + time: float = 0.25, + budget: float = 5.0, + verbose: bool = False, + quiet: bool = False, +) -> Tuple[bool, str, float, str, List[str]]: + """ + Attempt to fix prompts based on a single user story. + + This runs detect_change on the story, then applies changes to each affected + prompt by calling change_main with the story as the change prompt. + """ + from .change_main import change_main + + prompts_root = _resolve_prompts_dir(prompts_dir) + prompt_files = discover_prompt_files(str(prompts_root)) + + if not prompt_files: + msg = "No prompt files found to apply user story fixes." + return False, msg, 0.0, "", [] + + story_path = Path(story_file) + if not story_path.exists(): + return False, f"User story file not found: {story_file}", 0.0, "", [] + + story_content = _read_story(story_path) + changes_list, detect_cost, detect_model = detect_change( + [str(p) for p in prompt_files], + story_content, + strength, + temperature, + time, + verbose=verbose, + ) + + if not changes_list: + return True, "No prompt changes needed for this user story.", detect_cost, detect_model, [] + + total_cost = detect_cost + model_name = detect_model + changed_files: List[str] = [] + errors: List[str] = [] + + ctx_obj = getattr(ctx, "obj", None) or {} + original_skip = ctx_obj.get("skip_user_stories") + ctx_obj["skip_user_stories"] = True + setattr(ctx, "obj", ctx_obj) + + try: + for change in changes_list: + prompt_name = str(change.get("prompt_name") or "") + prompt_path = _resolve_prompt_path(prompt_name, prompt_files) + if not prompt_path: + errors.append(f"Unable to resolve prompt path: {prompt_name}") + continue + + code_path = _prompt_to_code_path(prompt_path, prompts_root) + if not code_path or not code_path.exists(): + errors.append(f"Code file not found for prompt: {prompt_path}") + continue + + result_message, cost, model = change_main( + ctx=ctx, + change_prompt_file=str(story_path), + input_code=str(code_path), + input_prompt_file=str(prompt_path), + output=None, + use_csv=False, + budget=budget, + ) + total_cost += cost + model_name = model or model_name + changed_files.append(str(prompt_path)) + if result_message.startswith("[bold red]Error"): + errors.append(result_message) + + finally: + if original_skip is None: + ctx_obj.pop("skip_user_stories", None) + else: + ctx_obj["skip_user_stories"] = original_skip + setattr(ctx, "obj", ctx_obj) + + if errors: + return False, "\n".join(errors), total_cost, model_name, changed_files + + # Re-run validation for just this story after applying changes + passed, _, validation_cost, validation_model = run_user_story_tests( + prompts_dir=str(prompts_root), + story_files=[story_path], + prompt_files=prompt_files, + strength=strength, + temperature=temperature, + time=time, + verbose=verbose, + quiet=quiet, + fail_fast=True, + ) + total_cost += validation_cost + if validation_model: + model_name = validation_model + + if not passed: + return False, "User story still failing after prompt updates.", total_cost, model_name, changed_files + + return True, "User story prompts updated successfully.", total_cost, model_name, changed_files diff --git a/tests/commands/test_analysis.py b/tests/commands/test_analysis.py index 6fed7a55..ed7f224e 100644 --- a/tests/commands/test_analysis.py +++ b/tests/commands/test_analysis.py @@ -2,7 +2,7 @@ import pytest from unittest.mock import patch, MagicMock from click.testing import CliRunner -from pdd.commands.analysis import detect_change, conflicts, bug, crash, trace +from pdd.commands.analysis import detect_change, conflicts, bug, crash, trace, story_test # ----------------------------------------------------------------------------- # Fixtures @@ -55,6 +55,41 @@ def test_detect_change_insufficient_args(runner, mock_context_obj): assert result.exit_code != 0 assert "Requires at least one PROMPT_FILE and one CHANGE_FILE" in result.output +# ----------------------------------------------------------------------------- +# Tests for 'story-test' command +# ----------------------------------------------------------------------------- + +def test_story_test_success(runner, mock_context_obj): + """Test 'story-test' command invokes runner with defaults.""" + with patch("pdd.commands.analysis.run_user_story_tests") as mock_runner: + mock_runner.return_value = (True, [{"story": "s", "passed": True}], 0.1, "gpt-4") + result = runner.invoke(story_test, [], obj=mock_context_obj) + + assert result.exit_code == 0 + args, kwargs = mock_runner.call_args + assert kwargs["prompts_dir"] is None + assert kwargs["stories_dir"] is None + assert kwargs["include_llm_prompts"] is False + assert kwargs["fail_fast"] is True + + +def test_story_test_options(runner, mock_context_obj): + """Test 'story-test' command forwards options.""" + with patch("pdd.commands.analysis.run_user_story_tests") as mock_runner: + mock_runner.return_value = (True, [], 0.0, "gpt-4") + result = runner.invoke( + story_test, + ["--stories-dir", "stories", "--prompts-dir", "prompts", "--include-llm", "--no-fail-fast"], + obj=mock_context_obj, + ) + + assert result.exit_code == 0 + args, kwargs = mock_runner.call_args + assert kwargs["prompts_dir"] == "prompts" + assert kwargs["stories_dir"] == "stories" + assert kwargs["include_llm_prompts"] is True + assert kwargs["fail_fast"] is False + # ----------------------------------------------------------------------------- # Tests for 'conflicts' command # ----------------------------------------------------------------------------- @@ -606,4 +641,4 @@ def test_trace_success(runner): mock_main.assert_called_once() kwargs = mock_main.call_args[1] assert kwargs["prompt_file"] == "prompt.txt" - assert kwargs["code_line"] == 42 \ No newline at end of file + assert kwargs["code_line"] == 42 diff --git a/tests/commands/test_fix.py b/tests/commands/test_fix.py index fabb8e72..f55c090a 100644 --- a/tests/commands/test_fix.py +++ b/tests/commands/test_fix.py @@ -198,4 +198,16 @@ def test_options_passing(runner, mock_deps): runner.invoke(fix, args) kwargs = mock_deps["fix_main"].call_args[1] assert kwargs["budget"] == 10.5 - assert kwargs["max_attempts"] == 7 \ No newline at end of file + + +def test_user_story_fix_mode(runner, mock_deps): + with patch("pdd.user_story_tests.run_user_story_fix") as mock_story_fix: + mock_story_fix.return_value = (True, "Story fixed", 0.2, "gpt-4", ["prompts/foo.prompt"]) + with runner.isolated_filesystem(): + with open("story__sample.md", "w") as fh: + fh.write("As a user...") + + result = runner.invoke(fix, ["story__sample.md"]) + + assert result.exit_code == 0 + mock_story_fix.assert_called_once() diff --git a/tests/test_change_main.py b/tests/test_change_main.py index 16a87fcf..ad2e5fc2 100644 --- a/tests/test_change_main.py +++ b/tests/test_change_main.py @@ -326,7 +326,9 @@ def test_change_main_non_csv_success( # or ensure the mock_open_function handles the Path object correctly. # Here, we pass the resolved path object directly to assert_called_once_with. with patch.object(Path, 'resolve', return_value=output_path_obj) as mock_resolve, \ - patch.object(Path, 'mkdir') as mock_mkdir: # Mock Path.mkdir + patch.object(Path, 'mkdir') as mock_mkdir, \ + patch("pdd.change_main.run_user_story_tests") as mock_story_tests: # Mock user story validation + mock_story_tests.return_value = (True, [], 0.0, "") result = change_main( ctx=ctx_instance, change_prompt_file=change_prompt_file, @@ -929,3 +931,894 @@ def test_change_csv_skips_empty_modified_content(mock_construct_paths, mock_proc empty_file = output_dir / "empty.prompt" assert not empty_file.exists(), \ f"File {empty_file} should not be written when modified_prompt is empty string" + + +def test_change_main_user_story_validation_failure(tmp_path): + change_file = tmp_path / "change.prompt" + code_file = tmp_path / "code.py" + prompt_file = tmp_path / "input.prompt" + output_file = tmp_path / "modified.prompt" + change_file.write_text("Change request", encoding="utf-8") + code_file.write_text("print('hi')", encoding="utf-8") + prompt_file.write_text("Original prompt", encoding="utf-8") + + ctx_instance = create_mock_context( + obj={ + "quiet": True, + "force": True, + "strength": DEFAULT_STRENGTH, + "temperature": 0, + "language": "python", + "extension": ".py", + "time": 0.25, + } + ) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.change_func") as mock_change, \ + patch("pdd.change_main.run_user_story_tests") as mock_story_tests: + mock_construct.return_value = ( + {"prompts_dir": str(tmp_path / "prompts")}, + { + "change_prompt_file": "Change request", + "input_code": "print('hi')", + "input_prompt_file": "Original prompt", + }, + {"output_prompt_file": str(output_file)}, + "python", + ) + mock_change.return_value = ("Modified prompt", 0.2, "model-a") + mock_story_tests.return_value = (False, [{"story": "s", "passed": False}], 0.3, "model-b") + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(code_file), + input_prompt_file=str(prompt_file), + output=str(output_file), + use_csv=False, + budget=5.0, + ) + + assert message == "User story validation failed. Review detect results for details." + assert total_cost == pytest.approx(0.5) + assert model_name == "model-a" + + +def test_change_main_skip_user_story_validation(tmp_path): + change_file = tmp_path / "change.prompt" + code_file = tmp_path / "code.py" + prompt_file = tmp_path / "input.prompt" + output_file = tmp_path / "modified.prompt" + change_file.write_text("Change request", encoding="utf-8") + code_file.write_text("print('hi')", encoding="utf-8") + prompt_file.write_text("Original prompt", encoding="utf-8") + + ctx_instance = create_mock_context( + obj={ + "quiet": True, + "force": True, + "skip_user_stories": True, + "strength": DEFAULT_STRENGTH, + "temperature": 0, + "language": "python", + "extension": ".py", + "time": 0.25, + } + ) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.change_func") as mock_change, \ + patch("pdd.change_main.run_user_story_tests") as mock_story_tests: + mock_construct.return_value = ( + {"prompts_dir": str(tmp_path / "prompts")}, + { + "change_prompt_file": "Change request", + "input_code": "print('hi')", + "input_prompt_file": "Original prompt", + }, + {"output_prompt_file": str(output_file)}, + "python", + ) + mock_change.return_value = ("Modified prompt", 0.2, "model-a") + mock_story_tests.side_effect = AssertionError("run_user_story_tests should not be called") + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(code_file), + input_prompt_file=str(prompt_file), + output=str(output_file), + use_csv=False, + budget=5.0, + ) + + assert "Modified prompt saved to" in message + assert total_cost == pytest.approx(0.2) + assert model_name == "model-a" + + +def test_change_main_user_story_validation_uses_output_dir(tmp_path): + csv_file = tmp_path / "changes.csv" + csv_file.write_text("prompt_name,change_instructions\nfoo.prompt,Do it\n", encoding="utf-8") + code_dir = tmp_path / "code" + code_dir.mkdir() + (code_dir / "foo.py").write_text("pass", encoding="utf-8") + output_dir = tmp_path / "modified_prompts" + + ctx_instance = create_mock_context( + obj={ + "quiet": True, + "force": True, + "strength": DEFAULT_STRENGTH, + "temperature": 0, + "language": "python", + "extension": ".py", + "time": 0.25, + } + ) + + out_prompt = output_dir / "foo.prompt" + base_prompt = tmp_path / "prompts" / "foo.prompt" + base_other = tmp_path / "prompts" / "bar.prompt" + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.process_csv_change") as mock_process, \ + patch("pdd.change_main.discover_prompt_files") as mock_discover, \ + patch("pdd.change_main.run_user_story_tests") as mock_story_tests: + mock_construct.return_value = ( + {"prompts_dir": str(tmp_path / "prompts")}, + {"change_prompt_file": csv_file.read_text(encoding="utf-8")}, + {"output_prompt_file": str(output_dir)}, + "python", + ) + mock_process.return_value = ( + True, + [{"file_name": "foo.prompt", "modified_prompt": "updated"}], + 0.1, + "model-a", + ) + mock_discover.side_effect = [[out_prompt], [base_prompt, base_other]] + mock_story_tests.return_value = (True, [], 0.0, "") + + change_main( + ctx=ctx_instance, + change_prompt_file=str(csv_file), + input_code=str(code_dir), + input_prompt_file=None, + output=str(output_dir), + use_csv=True, + budget=5.0, + ) + + _, kwargs = mock_story_tests.call_args + assert kwargs["prompt_files"] == [out_prompt, base_other] + + +def test_change_main_skips_user_story_validation_for_csv_output(tmp_path): + csv_file = tmp_path / "changes.csv" + csv_file.write_text("prompt_name,change_instructions\nfoo.prompt,Do it\n", encoding="utf-8") + code_dir = tmp_path / "code" + code_dir.mkdir() + (code_dir / "foo.py").write_text("pass", encoding="utf-8") + output_csv = tmp_path / "modified_prompts.csv" + + ctx_instance = create_mock_context( + obj={ + "quiet": True, + "force": True, + "strength": DEFAULT_STRENGTH, + "temperature": 0, + "language": "python", + "extension": ".py", + "time": 0.25, + } + ) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.process_csv_change") as mock_process, \ + patch("pdd.change_main.run_user_story_tests") as mock_story_tests: + mock_construct.return_value = ( + {"prompts_dir": str(tmp_path / "prompts")}, + {"change_prompt_file": csv_file.read_text(encoding="utf-8")}, + {"output_prompt_file": str(output_csv)}, + "python", + ) + mock_process.return_value = ( + True, + [{"file_name": "foo.prompt", "modified_prompt": "updated"}], + 0.1, + "model-a", + ) + mock_story_tests.side_effect = AssertionError("run_user_story_tests should not be called for CSV output") + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(csv_file), + input_code=str(code_dir), + input_prompt_file=None, + output=str(output_csv), + use_csv=True, + budget=5.0, + ) + + assert message == "Multiple prompts have been updated." + assert total_cost == pytest.approx(0.1) + assert model_name == "model-a" + + +def test_change_main_requires_change_prompt_and_input_code(): + ctx_instance = create_mock_context(obj={"quiet": True}) + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file="", + input_code="", + input_prompt_file=None, + output=None, + use_csv=False, + budget=5.0, + ) + + assert "Both --change-prompt-file and --input-code arguments are required" in message + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_csv_rejects_input_prompt_file(tmp_path): + csv_file = tmp_path / "changes.csv" + code_dir = tmp_path / "code" + csv_file.write_text("prompt_name,change_instructions\nfoo.prompt,Do it\n", encoding="utf-8") + code_dir.mkdir() + + ctx_instance = create_mock_context(obj={"quiet": True}) + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(csv_file), + input_code=str(code_dir), + input_prompt_file="should_not_be_set.prompt", + output=None, + use_csv=True, + budget=5.0, + ) + + assert "--input-prompt-file should not be provided when using --csv mode" in message + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_csv_empty_header_returns_error(tmp_path): + csv_file = tmp_path / "empty.csv" + code_dir = tmp_path / "code" + csv_file.write_text("", encoding="utf-8") + code_dir.mkdir() + + ctx_instance = create_mock_context(obj={"quiet": True}) + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(csv_file), + input_code=str(code_dir), + input_prompt_file=None, + output=None, + use_csv=True, + budget=5.0, + ) + + assert message.startswith("Failed to read or validate CSV header:") + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_csv_file_not_found(tmp_path): + code_dir = tmp_path / "code" + code_dir.mkdir() + missing_csv = tmp_path / "changes_input" + + ctx_instance = create_mock_context(obj={"quiet": True}) + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(missing_csv), + input_code=str(code_dir), + input_prompt_file=None, + output=None, + use_csv=True, + budget=5.0, + ) + + assert message == f"CSV file not found: {missing_csv}" + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_csv_read_exception_returns_error(tmp_path): + csv_file = tmp_path / "changes.csv" + code_dir = tmp_path / "code" + csv_file.write_text("prompt_name,change_instructions\nfoo.prompt,Do it\n", encoding="utf-8") + code_dir.mkdir() + + ctx_instance = create_mock_context(obj={"quiet": True}) + + with patch("builtins.open", side_effect=PermissionError("denied")): + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(csv_file), + input_code=str(code_dir), + input_prompt_file=None, + output=None, + use_csv=True, + budget=5.0, + ) + + assert message.startswith(f"Failed to open or read CSV file '{csv_file}':") + assert "denied" in message + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_non_csv_rejects_directory_input_code(tmp_path): + change_file = tmp_path / "change.prompt" + prompt_file = tmp_path / "input.prompt" + input_code_dir = tmp_path / "src" + change_file.write_text("change", encoding="utf-8") + prompt_file.write_text("prompt", encoding="utf-8") + input_code_dir.mkdir() + + ctx_instance = create_mock_context(obj={"quiet": True}) + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(input_code_dir), + input_prompt_file=str(prompt_file), + output=None, + use_csv=False, + budget=5.0, + ) + + assert "must be a file path, not a directory" in message + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_csv_extension_resolution_error(tmp_path): + csv_file = tmp_path / "changes.csv" + code_dir = tmp_path / "code" + csv_file.write_text("prompt_name,change_instructions\nfoo.prompt,Do it\n", encoding="utf-8") + code_dir.mkdir() + + ctx_instance = create_mock_context( + obj={ + "quiet": True, + "language": "unknownlang", + "extension": None, + } + ) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config") as mock_config, \ + patch("pdd.change_main.get_extension", side_effect=ValueError("unsupported")): + mock_construct.return_value = ({}, {"change_prompt_file": "csv"}, {}, "unknownlang") + mock_config.return_value = {"strength": 0.2, "temperature": 0.0, "time": 0.25} + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(csv_file), + input_code=str(code_dir), + input_prompt_file=None, + output=None, + use_csv=True, + budget=5.0, + ) + + assert "Could not determine file extension for language" in message + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_csv_process_exception_returns_default_message(tmp_path): + csv_file = tmp_path / "changes.csv" + code_dir = tmp_path / "code" + csv_file.write_text("prompt_name,change_instructions\nfoo.prompt,Do it\n", encoding="utf-8") + code_dir.mkdir() + + ctx_instance = create_mock_context(obj={"quiet": True}) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config") as mock_config, \ + patch("pdd.change_main.process_csv_change", side_effect=RuntimeError("csv boom")): + mock_construct.return_value = ({}, {"change_prompt_file": "csv"}, {}, "python") + mock_config.return_value = {"strength": 0.2, "temperature": 0.0, "time": 0.25} + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(csv_file), + input_code=str(code_dir), + input_prompt_file=None, + output=None, + use_csv=True, + budget=5.0, + ) + + assert message == "Multiple prompts have been updated." + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_non_csv_missing_input_content(tmp_path): + change_file = tmp_path / "change.prompt" + prompt_file = tmp_path / "input.prompt" + code_file = tmp_path / "code.py" + output_file = tmp_path / "output.prompt" + change_file.write_text("change", encoding="utf-8") + prompt_file.write_text("prompt", encoding="utf-8") + code_file.write_text("print('x')", encoding="utf-8") + + ctx_instance = create_mock_context(obj={"quiet": True}) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config") as mock_config: + mock_construct.return_value = ( + {}, + { + "change_prompt_file": "change", + "input_code": "", + "input_prompt_file": "prompt", + }, + {"output_prompt_file": str(output_file)}, + "python", + ) + mock_config.return_value = {"strength": 0.2, "temperature": 0.0, "time": 0.25} + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(code_file), + input_prompt_file=str(prompt_file), + output=str(output_file), + use_csv=False, + budget=5.0, + ) + + assert message.startswith("Failed to read content for required input files:") + assert "input_code" in message + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_non_csv_uses_construct_paths_default_output(tmp_path): + change_file = tmp_path / "change.prompt" + prompt_file = tmp_path / "input.prompt" + code_file = tmp_path / "code.py" + default_output_file = tmp_path / "default_output.prompt" + change_file.write_text("change", encoding="utf-8") + prompt_file.write_text("prompt", encoding="utf-8") + code_file.write_text("print('x')", encoding="utf-8") + + ctx_instance = create_mock_context(obj={"quiet": True, "skip_user_stories": True}) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config") as mock_config, \ + patch("pdd.change_main.change_func") as mock_change: + mock_construct.return_value = ( + {"prompts_dir": str(tmp_path / "prompts")}, + { + "change_prompt_file": "change", + "input_code": "print('x')", + "input_prompt_file": "prompt", + }, + {"output_prompt_file": str(default_output_file)}, + "python", + ) + mock_config.return_value = {"strength": 0.2, "temperature": 0.0, "time": 0.25} + mock_change.return_value = ("updated prompt text", 0.2, "model-x") + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(code_file), + input_prompt_file=str(prompt_file), + output=None, + use_csv=False, + budget=5.0, + ) + + assert message == f"Modified prompt saved to {default_output_file.resolve()}" + assert default_output_file.read_text(encoding="utf-8") == "updated prompt text" + assert total_cost == pytest.approx(0.2) + assert model_name == "model-x" + + +def test_change_main_non_csv_missing_output_path(tmp_path): + change_file = tmp_path / "change.prompt" + prompt_file = tmp_path / "input.prompt" + code_file = tmp_path / "code.py" + change_file.write_text("change", encoding="utf-8") + prompt_file.write_text("prompt", encoding="utf-8") + code_file.write_text("print('x')", encoding="utf-8") + + ctx_instance = create_mock_context(obj={"quiet": True, "skip_user_stories": True}) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config") as mock_config, \ + patch("pdd.change_main.change_func") as mock_change: + mock_construct.return_value = ( + {}, + { + "change_prompt_file": "change", + "input_code": "print('x')", + "input_prompt_file": "prompt", + }, + {}, + "python", + ) + mock_config.return_value = {"strength": 0.2, "temperature": 0.0, "time": 0.25} + mock_change.return_value = ("updated prompt text", 0.2, "model-x") + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(code_file), + input_prompt_file=str(prompt_file), + output=None, + use_csv=False, + budget=5.0, + ) + + assert message == "Could not determine output path for modified prompt." + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_non_csv_write_io_error(tmp_path): + change_file = tmp_path / "change.prompt" + prompt_file = tmp_path / "input.prompt" + code_file = tmp_path / "code.py" + output_file = tmp_path / "output.prompt" + change_file.write_text("change", encoding="utf-8") + prompt_file.write_text("prompt", encoding="utf-8") + code_file.write_text("print('x')", encoding="utf-8") + + ctx_instance = create_mock_context(obj={"quiet": True, "skip_user_stories": True}) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config") as mock_config, \ + patch("pdd.change_main.change_func") as mock_change, \ + patch("builtins.open", side_effect=IOError("disk full")): + mock_construct.return_value = ( + {}, + { + "change_prompt_file": "change", + "input_code": "print('x')", + "input_prompt_file": "prompt", + }, + {"output_prompt_file": str(output_file)}, + "python", + ) + mock_config.return_value = {"strength": 0.2, "temperature": 0.0, "time": 0.25} + mock_change.return_value = ("updated prompt text", 0.2, "model-x") + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(code_file), + input_prompt_file=str(prompt_file), + output=str(output_file), + use_csv=False, + budget=5.0, + ) + + assert message.startswith("Failed to write output file") + assert total_cost == pytest.approx(0.2) + assert model_name == "model-x" + + +def test_change_main_non_csv_write_unexpected_error(tmp_path): + change_file = tmp_path / "change.prompt" + prompt_file = tmp_path / "input.prompt" + code_file = tmp_path / "code.py" + output_file = tmp_path / "output.prompt" + change_file.write_text("change", encoding="utf-8") + prompt_file.write_text("prompt", encoding="utf-8") + code_file.write_text("print('x')", encoding="utf-8") + + ctx_instance = create_mock_context(obj={"quiet": True, "skip_user_stories": True}) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config") as mock_config, \ + patch("pdd.change_main.change_func") as mock_change, \ + patch("builtins.open", side_effect=RuntimeError("boom")): + mock_construct.return_value = ( + {}, + { + "change_prompt_file": "change", + "input_code": "print('x')", + "input_prompt_file": "prompt", + }, + {"output_prompt_file": str(output_file)}, + "python", + ) + mock_config.return_value = {"strength": 0.2, "temperature": 0.0, "time": 0.25} + mock_change.return_value = ("updated prompt text", 0.2, "model-x") + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(code_file), + input_prompt_file=str(prompt_file), + output=str(output_file), + use_csv=False, + budget=5.0, + ) + + assert message.startswith("Unexpected error writing output file") + assert total_cost == pytest.approx(0.2) + assert model_name == "model-x" + + +def test_change_main_handles_top_level_file_not_found(tmp_path): + change_file = tmp_path / "change.prompt" + prompt_file = tmp_path / "input.prompt" + code_file = tmp_path / "code.py" + output_file = tmp_path / "output.prompt" + change_file.write_text("change", encoding="utf-8") + prompt_file.write_text("prompt", encoding="utf-8") + code_file.write_text("print('x')", encoding="utf-8") + + ctx_instance = create_mock_context(obj={"quiet": True}) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config", side_effect=FileNotFoundError("missing.cfg")): + mock_construct.return_value = ( + {}, + { + "change_prompt_file": "change", + "input_code": "print('x')", + "input_prompt_file": "prompt", + }, + {"output_prompt_file": str(output_file)}, + "python", + ) + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(code_file), + input_prompt_file=str(prompt_file), + output=str(output_file), + use_csv=False, + budget=5.0, + ) + + assert message.startswith("Input file not found:") + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_handles_top_level_not_a_directory(tmp_path): + change_file = tmp_path / "change.prompt" + prompt_file = tmp_path / "input.prompt" + code_file = tmp_path / "code.py" + output_file = tmp_path / "output.prompt" + change_file.write_text("change", encoding="utf-8") + prompt_file.write_text("prompt", encoding="utf-8") + code_file.write_text("print('x')", encoding="utf-8") + + ctx_instance = create_mock_context(obj={"quiet": True}) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config", side_effect=NotADirectoryError("bad dir")): + mock_construct.return_value = ( + {}, + { + "change_prompt_file": "change", + "input_code": "print('x')", + "input_prompt_file": "prompt", + }, + {"output_prompt_file": str(output_file)}, + "python", + ) + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(code_file), + input_prompt_file=str(prompt_file), + output=str(output_file), + use_csv=False, + budget=5.0, + ) + + assert message.startswith("Expected a directory but found a file, or vice versa:") + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_handles_top_level_unexpected_error(tmp_path): + change_file = tmp_path / "change.prompt" + prompt_file = tmp_path / "input.prompt" + code_file = tmp_path / "code.py" + output_file = tmp_path / "output.prompt" + change_file.write_text("change", encoding="utf-8") + prompt_file.write_text("prompt", encoding="utf-8") + code_file.write_text("print('x')", encoding="utf-8") + + ctx_instance = create_mock_context(obj={"quiet": True}) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config", side_effect=RuntimeError("kaboom")): + mock_construct.return_value = ( + {}, + { + "change_prompt_file": "change", + "input_code": "print('x')", + "input_prompt_file": "prompt", + }, + {"output_prompt_file": str(output_file)}, + "python", + ) + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(code_file), + input_prompt_file=str(prompt_file), + output=str(output_file), + use_csv=False, + budget=5.0, + ) + + assert message == "An unexpected error occurred: kaboom" + assert total_cost == 0.0 + assert model_name == "" + + +def test_change_main_csv_skips_malformed_output_rows(tmp_path): + csv_file = tmp_path / "changes.csv" + output_csv = tmp_path / "result.csv" + code_dir = tmp_path / "code" + csv_file.write_text("prompt_name,change_instructions\nfoo.prompt,Do it\n", encoding="utf-8") + code_dir.mkdir() + + ctx_instance = create_mock_context(obj={"quiet": True, "skip_user_stories": True}) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config") as mock_config, \ + patch("pdd.change_main.process_csv_change") as mock_process: + mock_construct.return_value = ( + {"prompts_dir": str(tmp_path / "prompts")}, + {"change_prompt_file": csv_file.read_text(encoding="utf-8")}, + {"output_prompt_file": str(output_csv)}, + "python", + ) + mock_config.return_value = {"strength": 0.2, "temperature": 0.0, "time": 0.25} + mock_process.return_value = ( + True, + [ + {"file_name": "ok.prompt", "modified_prompt": "ok content"}, + {"file_name": "bad.prompt", "modified_prompt": None}, + ], + 0.1, + "model-x", + ) + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(csv_file), + input_code=str(code_dir), + input_prompt_file=None, + output=str(output_csv), + use_csv=True, + budget=5.0, + ) + + output_text = output_csv.read_text(encoding="utf-8") + assert "ok.prompt" in output_text + assert "bad.prompt" not in output_text + assert message == "Multiple prompts have been updated." + assert total_cost == pytest.approx(0.1) + assert model_name == "model-x" + + +def test_change_main_csv_output_write_io_error(tmp_path): + csv_file = tmp_path / "changes.csv" + output_csv = tmp_path / "result.csv" + code_dir = tmp_path / "code" + csv_file.write_text("prompt_name,change_instructions\nfoo.prompt,Do it\n", encoding="utf-8") + code_dir.mkdir() + + ctx_instance = create_mock_context(obj={"quiet": True, "skip_user_stories": True}) + real_open = open + + def open_side_effect(file, mode="r", *args, **kwargs): + if mode.startswith("w") and Path(file) == output_csv.resolve(): + raise IOError("cannot write") + return real_open(file, mode, *args, **kwargs) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.resolve_effective_config") as mock_config, \ + patch("pdd.change_main.process_csv_change") as mock_process, \ + patch("builtins.open", side_effect=open_side_effect): + mock_construct.return_value = ( + {"prompts_dir": str(tmp_path / "prompts")}, + {"change_prompt_file": csv_file.read_text(encoding="utf-8")}, + {"output_prompt_file": str(output_csv)}, + "python", + ) + mock_config.return_value = {"strength": 0.2, "temperature": 0.0, "time": 0.25} + mock_process.return_value = ( + True, + [{"file_name": "ok.prompt", "modified_prompt": "ok content"}], + 0.3, + "model-x", + ) + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(csv_file), + input_code=str(code_dir), + input_prompt_file=None, + output=str(output_csv), + use_csv=True, + budget=5.0, + ) + + assert message == "Multiple prompts have been updated." + assert total_cost == pytest.approx(0.3) + assert model_name == "model-x" + + +def test_change_main_story_validation_failure_prints_error_when_not_quiet(tmp_path): + change_file = tmp_path / "change.prompt" + code_file = tmp_path / "code.py" + prompt_file = tmp_path / "input.prompt" + output_file = tmp_path / "modified.prompt" + change_file.write_text("Change request", encoding="utf-8") + code_file.write_text("print('hi')", encoding="utf-8") + prompt_file.write_text("Original prompt", encoding="utf-8") + + ctx_instance = create_mock_context( + obj={ + "quiet": False, + "force": True, + "strength": DEFAULT_STRENGTH, + "temperature": 0, + "language": "python", + "extension": ".py", + "time": 0.25, + } + ) + + with patch("pdd.change_main.construct_paths") as mock_construct, \ + patch("pdd.change_main.change_func") as mock_change, \ + patch("pdd.change_main.run_user_story_tests") as mock_story_tests, \ + patch("pdd.change_main.rprint") as mock_rprint: + mock_construct.return_value = ( + {"prompts_dir": str(tmp_path / "prompts")}, + { + "change_prompt_file": "Change request", + "input_code": "print('hi')", + "input_prompt_file": "Original prompt", + }, + {"output_prompt_file": str(output_file)}, + "python", + ) + mock_change.return_value = ("Modified prompt", 0.2, "model-a") + mock_story_tests.return_value = (False, [{"story": "s", "passed": False}], 0.3, "model-b") + + message, total_cost, model_name = change_main( + ctx=ctx_instance, + change_prompt_file=str(change_file), + input_code=str(code_file), + input_prompt_file=str(prompt_file), + output=str(output_file), + use_csv=False, + budget=5.0, + ) + + assert message == "User story validation failed. Review detect results for details." + assert total_cost == pytest.approx(0.5) + assert model_name == "model-a" + mock_rprint.assert_any_call("[bold red]Error:[/bold red] User story validation failed. Review detect results for details.") diff --git a/tests/test_user_story_tests.py b/tests/test_user_story_tests.py new file mode 100644 index 00000000..f7e12063 --- /dev/null +++ b/tests/test_user_story_tests.py @@ -0,0 +1,185 @@ +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import patch + +import pytest + +from pdd.user_story_tests import ( + discover_prompt_files, + discover_story_files, + run_user_story_fix, + run_user_story_tests, +) + + +def test_user_story_tests_no_stories(tmp_path): + prompts_dir = tmp_path / "prompts" + prompts_dir.mkdir() + (prompts_dir / "foo_python.prompt").write_text("prompt", encoding="utf-8") + + passed, results, cost, model = run_user_story_tests( + prompts_dir=str(prompts_dir), + stories_dir=str(tmp_path / "user_stories"), + quiet=True, + ) + + assert passed is True + assert results == [] + assert cost == 0.0 + assert model == "" + + +def test_user_story_tests_detect_pass(tmp_path): + prompts_dir = tmp_path / "prompts" + stories_dir = tmp_path / "user_stories" + prompts_dir.mkdir() + stories_dir.mkdir() + + (prompts_dir / "foo_python.prompt").write_text("prompt", encoding="utf-8") + story = stories_dir / "story__happy_path.md" + story.write_text("As a user...", encoding="utf-8") + + with patch("pdd.user_story_tests.detect_change") as mock_detect: + mock_detect.return_value = ([], 0.25, "gpt-test") + passed, results, cost, model = run_user_story_tests( + prompts_dir=str(prompts_dir), + stories_dir=str(stories_dir), + quiet=True, + ) + + assert passed is True + assert results[0]["passed"] is True + assert cost == 0.25 + assert model == "gpt-test" + + +def test_user_story_tests_detect_fail(tmp_path): + prompts_dir = tmp_path / "prompts" + stories_dir = tmp_path / "user_stories" + prompts_dir.mkdir() + stories_dir.mkdir() + + (prompts_dir / "foo_python.prompt").write_text("prompt", encoding="utf-8") + story = stories_dir / "story__failure.md" + story.write_text("As a user...", encoding="utf-8") + + changes = [{"prompt_name": "foo_python.prompt", "change_instructions": "Add support"}] + + with patch("pdd.user_story_tests.detect_change") as mock_detect: + mock_detect.return_value = (changes, 0.5, "gpt-test") + passed, results, cost, model = run_user_story_tests( + prompts_dir=str(prompts_dir), + stories_dir=str(stories_dir), + quiet=True, + ) + + assert passed is False + assert results[0]["passed"] is False + assert results[0]["changes"] == changes + assert cost == 0.5 + assert model == "gpt-test" + + +def test_discover_prompt_files_excludes_llm_by_default(tmp_path): + prompts_dir = tmp_path / "prompts" + prompts_dir.mkdir() + (prompts_dir / "foo_python.prompt").write_text("prompt", encoding="utf-8") + (prompts_dir / "bar_llm.prompt").write_text("prompt", encoding="utf-8") + + results = discover_prompt_files(str(prompts_dir)) + + assert len(results) == 1 + assert results[0].name == "foo_python.prompt" + + +def test_discover_prompt_files_includes_llm(tmp_path): + prompts_dir = tmp_path / "prompts" + prompts_dir.mkdir() + (prompts_dir / "foo_python.prompt").write_text("prompt", encoding="utf-8") + (prompts_dir / "bar_llm.prompt").write_text("prompt", encoding="utf-8") + + results = discover_prompt_files(str(prompts_dir), include_llm=True) + + assert {p.name for p in results} == {"foo_python.prompt", "bar_llm.prompt"} + + +def test_discover_story_files_filters_prefix(tmp_path): + stories_dir = tmp_path / "user_stories" + stories_dir.mkdir() + (stories_dir / "story__one.md").write_text("story", encoding="utf-8") + (stories_dir / "not_a_story.md").write_text("story", encoding="utf-8") + + results = discover_story_files(str(stories_dir)) + + assert [p.name for p in results] == ["story__one.md"] + + +def test_user_story_tests_fail_fast(tmp_path): + prompts_dir = tmp_path / "prompts" + stories_dir = tmp_path / "user_stories" + prompts_dir.mkdir() + stories_dir.mkdir() + + (prompts_dir / "foo_python.prompt").write_text("prompt", encoding="utf-8") + (stories_dir / "story__one.md").write_text("story", encoding="utf-8") + (stories_dir / "story__two.md").write_text("story", encoding="utf-8") + + changes = [{"prompt_name": "foo_python.prompt", "change_instructions": "Add support"}] + + with patch("pdd.user_story_tests.detect_change") as mock_detect: + mock_detect.return_value = (changes, 0.5, "gpt-test") + passed, results, cost, model = run_user_story_tests( + prompts_dir=str(prompts_dir), + stories_dir=str(stories_dir), + quiet=True, + fail_fast=True, + ) + + assert passed is False + assert len(results) == 1 + assert mock_detect.call_count == 1 + assert cost == 0.5 + assert model == "gpt-test" + + +def test_user_story_fix_happy_path(tmp_path): + prompts_dir = tmp_path / "prompts" / "sub" + src_dir = tmp_path / "src" / "sub" + stories_dir = tmp_path / "user_stories" + prompts_dir.mkdir(parents=True) + src_dir.mkdir(parents=True) + stories_dir.mkdir() + + prompt_path = prompts_dir / "calc_python.prompt" + prompt_path.write_text("prompt", encoding="utf-8") + code_path = src_dir / "calc.py" + code_path.write_text("code", encoding="utf-8") + story_path = stories_dir / "story__calc.md" + story_path.write_text("story", encoding="utf-8") + + ctx = SimpleNamespace(obj={}) + + with patch("pdd.user_story_tests.discover_prompt_files") as mock_discover, \ + patch("pdd.user_story_tests.detect_change") as mock_detect, \ + patch("pdd.change_main.change_main") as mock_change, \ + patch("pdd.user_story_tests.run_user_story_tests") as mock_story_tests: + mock_discover.return_value = [prompt_path] + mock_detect.return_value = ([{"prompt_name": "calc_python.prompt"}], 0.1, "detect-model") + mock_change.return_value = ("ok", 0.2, "change-model") + mock_story_tests.return_value = (True, [], 0.3, "verify-model") + + success, message, cost, model, changed_files = run_user_story_fix( + ctx=ctx, + story_file=str(story_path), + prompts_dir=str(tmp_path / "prompts"), + quiet=True, + ) + + assert success is True, message + assert message == "User story prompts updated successfully." + assert cost == pytest.approx(0.6) + assert model == "verify-model" + assert changed_files == [str(prompt_path)] + assert ctx.obj.get("skip_user_stories") is None + mock_change.assert_called_once() + assert mock_change.call_args[1]["input_code"] == str(code_path) diff --git a/user_stories/story__template.md b/user_stories/story__template.md new file mode 100644 index 00000000..30c85990 --- /dev/null +++ b/user_stories/story__template.md @@ -0,0 +1,17 @@ +# User Story: + +## Story +As a , I want so that . + +## Context +Describe any relevant context, constraints, or assumptions. + +## Acceptance Criteria +1. Given , when , then . +2. Given , when , then . + +## Non-Goals +1. What this story explicitly does not cover. + +## Notes +- Links, edge cases, or implementation hints that help interpret intent.