Skip to content

Add Generated Artifact Verification to reflection framework #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rhart696 wants to merge 1 commit into
base: master
Choose a base branch
from
+66 −2
Open

Add Generated Artifact Verification to reflection framework #36

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 66 additions & 2 deletions plugins/reflexion/commands/reflect.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,60 @@ Before proceeding, evaluate your most recent output against these criteria:
- [ ] Are there edge cases that haven't been considered?
- [ ] Could there be unintended side effects?

4. **Fact-Checking Required**
4. **Dependency & Impact Verification** (CRITICAL - per ISSUE-086, DEC-096)
- [ ] For ANY proposed addition/deletion/modification, have you checked for dependencies?
- [ ] Have you searched for related decisions (DEC-###) that may be superseded or supersede this?
- [ ] Have you checked AUTHORITATIVE.yaml for active evaluations or status?
- [ ] Have you searched the ecosystem for files/processes that depend on items being changed?
- [ ] If recommending removal of anything, have you verified nothing depends on it?

**Mandatory Checks Before Recommending Changes:**
```bash
# Check for active evaluations/status
grep -A20 "item_name" ~/dev/AUTHORITATIVE.yaml | grep -i "status\|evaluation\|active"

# Check for ecosystem dependencies
grep -ri "item_name" ~/dev/infrastructure/ --include="*.md" --include="*.yaml" | head -20

# Check for related/superseding decisions
grep -i "item_name" ~/dev/infrastructure/dev-env-docs/DECISIONS-LOG.md | head -10

# Check for dedicated project directories
find ~/dev/infrastructure -maxdepth 2 -type d -iname "*item_name*" 2>/dev/null
```

**HARD RULE:** If ANY check reveals active dependencies, evaluations, or pending decisions, FLAG THIS IN THE EVALUATION. Do not approve work that recommends changes without dependency verification.

5. **Fact-Checking Required**
- [ ] Have you made any claims about performance? (needs verification)
- [ ] Have you stated any technical facts? (needs source/verification)
- [ ] Have you referenced best practices? (needs validation)
- [ ] Have you made security assertions? (needs careful review)

6. **Generated Artifact Verification** (CRITICAL for any generated code/content)
- [ ] **Cross-references validated**: Any references to external tools, APIs, or files verified to exist with correct names
- [ ] **Security scan**: Generated files checked for sensitive information (absolute paths with usernames, credentials, internal URLs)
- [ ] **Documentation sync**: If counts, stats, or references changed, all documentation citing them updated
- [ ] **State verification**: Claims about system state verified with actual commands, not memory

**Verification Commands (run before declaring complete):**
```bash
# Cross-reference check: verify tool/API names exist
# Example for MCP tools:
grep -o 'mcp_[a-z_]*' generated_file.py | sort -u | while read tool; do
grep -q "$tool" ~/.config/claude/claude_desktop_config.json || echo "MISSING: $tool"
done

# Security scan: check staged files for sensitive paths (Linux, macOS, Windows)
git diff --cached --name-only | xargs grep -l '/home/\|/Users/\|C:\\Users\|%USERPROFILE%' 2>/dev/null
Copy link

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The git diff --cached --name-only | xargs grep -l ... pipeline will emit an error (grep: missing file operand) and a non-zero exit status when there are no staged files, which is a common legitimate case and conflicts with the "HARD RULE" expectation that these commands can be run cleanly as a gate. Consider making the command no-op-safe for an empty file list (for example by using an option like xargs -r or guarding the grep invocation) so that the verification step doesn't fail spuriously when there's nothing to scan.

Suggested change
git diff --cached --name-only | xargs grep -l '/home/\|/Users/\|C:\\Users\|%USERPROFILE%' 2>/dev/null
git diff --cached --name-only | xargs -r grep -l '/home/\|/Users/\|C:\\Users\|%USERPROFILE%' 2>/dev/null

Copilot uses AI. Check for mistakes.

# Documentation sync: find docs referencing old values after changes
# Example: if you changed a count from 117 to 118
grep -rn "117" docs/ *.md | grep -i "count\|total\|items"
```

**HARD RULE:** Do not declare work complete until verification commands confirm claims match reality.

### Step 2: Decision Point

Based on the assessment above, determine:
Expand Down Expand Up @@ -526,7 +574,18 @@ Automatically trigger refinement if any of these conditions are met:
- No library search for common problems
- No consideration of existing services

4. **Architecture Violations**
4. **Dependency/Impact Gaps** (CRITICAL)
- Recommended deletion/removal without dependency check
- Cited prior decision (DEC-###) without checking for superseding decisions
- Proposed config changes without checking AUTHORITATIVE.yaml
- Modified ecosystem files without searching for dependents
- Any destructive action without PRE-MODIFICATION GATE checks
Copy link

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reference to PRE-MODIFICATION GATE here introduces a critical-sounding check without explaining what that gate is or where its checks are defined, which can make it hard for readers to understand how to comply with this trigger. Consider either defining the PRE-MODIFICATION GATE earlier in this document or linking to the section/file where its checklist and process are described, so evaluators have concrete guidance on what is expected.

Suggested change
- Any destructive action without PRE-MODIFICATION GATE checks
- Any destructive action without passing the PRE-MODIFICATION GATE (a pre-change safety checklist covering dependencies, backups, approvals, and rollback)

Copilot uses AI. Check for mistakes.
- Generated cross-references without validation against source of truth
- Committed files containing absolute paths or usernames
- Changed counts/stats without updating referencing documentation
- Declared complete without running verification commands

5. **Architecture Violations**
- Business logic in controllers/views
- Domain logic depending on infrastructure
- Unclear boundaries between contexts
Expand All @@ -548,6 +607,11 @@ Before finalizing any output:
- [ ] Did I search for existing libraries before writing custom code?
- [ ] Is the architecture aligned with Clean Architecture/DDD principles?
- [ ] Are names domain-specific rather than generic (utils/helpers)?
- [ ] **CROSS-REFERENCE CHECK:** Any tool/API/file references verified against actual inventory (not assumed)
- [ ] **SECURITY CHECK:** Generated files scanned for sensitive info (paths, usernames, credentials)
- [ ] **DOCUMENTATION SYNC:** All docs referencing changed values have been updated
- [ ] **STATE VERIFICATION:** Claims verified with actual commands, not memory
- [ ] **DEPENDENCY CHECK:** For any additions/deletions/modifications, have I verified no active dependencies, evaluations, or superseding decisions exist?

### Reflexion Questions

Expand Down
Loading