-
Notifications
You must be signed in to change notification settings - Fork 30
Add Generated Artifact Verification to reflection framework #36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Add Generated Artifact Verification to reflection framework #36
Conversation
Add verification steps for generated artifacts before declaring work complete. This catches common AI agent failures: - Cross-references to non-existent tools/APIs - Sensitive information in committed files (absolute paths, usernames) - Documentation drift (stale counts, outdated references) - Claims not verified against actual system state Changes: - Add Step 1.6: Generated Artifact Verification with checklist and commands - Add 4 new items to Refinement Triggers (Dependency/Impact Gaps) - Add 4 new items to Self-Refine Checklist Motivation: External review consistently catches issues that self-reflection misses. These verification steps formalize what external reviewers check. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR extends the reflection framework to explicitly verify generated artifacts (code/content) and detect common AI agent failure modes before work is marked complete.
Changes:
- Add a new Dependency & Impact Verification section with mandatory CLI checks for dependencies, decisions, and ecosystem impact.
- Introduce a Generated Artifact Verification checklist plus concrete verification commands for cross-references, security scanning, and documentation sync.
- Expand refinement triggers and the final self-refine checklist to cover dependency/impact gaps and generated artifact verification requirements.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| done | ||
|
|
||
| # Security scan: check staged files for sensitive paths (Linux, macOS, Windows) | ||
| git diff --cached --name-only | xargs grep -l '/home/\|/Users/\|C:\\Users\|%USERPROFILE%' 2>/dev/null |
Copilot
AI
Jan 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The git diff --cached --name-only | xargs grep -l ... pipeline will emit an error (grep: missing file operand) and a non-zero exit status when there are no staged files, which is a common legitimate case and conflicts with the "HARD RULE" expectation that these commands can be run cleanly as a gate. Consider making the command no-op-safe for an empty file list (for example by using an option like xargs -r or guarding the grep invocation) so that the verification step doesn't fail spuriously when there's nothing to scan.
| git diff --cached --name-only | xargs grep -l '/home/\|/Users/\|C:\\Users\|%USERPROFILE%' 2>/dev/null | |
| git diff --cached --name-only | xargs -r grep -l '/home/\|/Users/\|C:\\Users\|%USERPROFILE%' 2>/dev/null |
| - Cited prior decision (DEC-###) without checking for superseding decisions | ||
| - Proposed config changes without checking AUTHORITATIVE.yaml | ||
| - Modified ecosystem files without searching for dependents | ||
| - Any destructive action without PRE-MODIFICATION GATE checks |
Copilot
AI
Jan 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reference to PRE-MODIFICATION GATE here introduces a critical-sounding check without explaining what that gate is or where its checks are defined, which can make it hard for readers to understand how to comply with this trigger. Consider either defining the PRE-MODIFICATION GATE earlier in this document or linking to the section/file where its checklist and process are described, so evaluators have concrete guidance on what is expected.
| - Any destructive action without PRE-MODIFICATION GATE checks | |
| - Any destructive action without passing the PRE-MODIFICATION GATE (a pre-change safety checklist covering dependencies, backups, approvals, and rollback) |
What
Add "Generated Artifact Verification" step to the reflection framework, catching common AI agent failures before work is declared complete.
Why
External code review consistently catches issues that self-reflection misses:
query_decisioninstead ofquery_decisions/home/username/...in committed filesChanges
Token Impact
Adds ~400 tokens to ~6500 existing (~6% increase).
Testing Status
Motivation source: Real-world scenario where external review (Codex GPT-5.2) caught 4 issues that self-reflection (Claude + /reflexion:reflect at 3.5/5 score) missed.
Testing recommendation: Apply changes locally with
--plugin-dirand verify new checklist items appear during reflection.Checklist
Note
Introduces critical verification gates to the reflection checklist in
plugins/reflexion/commands/reflect.md.AUTHORITATIVE.yaml, decisions logs, and ecosystem dependents; includes a hard rule to flag active dependenciesWritten by Cursor Bugbot for commit 988a7ea. This will update automatically on new commits. Configure here.