Skip to content

feat(skills): add ce-audit skill for knowledge freshness auditing#242

Open
mvanhorn wants to merge 2 commits intoEveryInc:mainfrom
mvanhorn:feat/ce-audit-learnings
Open

feat(skills): add ce-audit skill for knowledge freshness auditing#242
mvanhorn wants to merge 2 commits intoEveryInc:mainfrom
mvanhorn:feat/ce-audit-learnings

Conversation

@mvanhorn
Copy link
Contributor

@mvanhorn mvanhorn commented Mar 9, 2026

Summary

Adds /ce:audit skill that detects stale learnings in docs/solutions/ by cross-referencing documented solutions against the current codebase.

Problem

The compound engineering cycle writes learnings via /ce:compound and retrieves them via learnings-researcher. But nothing detects when learnings go stale. Over time, refactors, migrations, and feature changes invalidate documented solutions that persist as if still valid.

This is a real problem users are hitting:

The compound philosophy says "each unit of work should make subsequent work easier." Stale learnings do the opposite - they recommend patterns that no longer apply, reference files that have moved, and suggest fixes for problems that have been restructured away.

Competitive context: No competing plugin (superpowers, deep-plan, everything-claude-code, claude-code-workflows) addresses learning decay. The everything-claude-code project has "confidence scoring" on learnings, and Windsurf has persistent memory with Flows - both signals that freshness management is the natural next evolution for compound engineering's core moat (the compound step).

How It Works

The skill runs a 6-step process:

  1. Discover all markdown files in docs/solutions/ with YAML frontmatter
  2. Extract references from each learning: file paths, module/class names, component references, code identifiers, dates
  3. Cross-reference against the codebase using 5 staleness signals:
Signal Weight Detection Method
Referenced file deleted High test -f path
Module/class not found in codebase High Grep for class/module name
Referenced file heavily modified Medium git log --since=LEARNING_DATE --stat path
Parent directory restructured Medium git log --diff-filter=R for renames
Age > 90 days with no validation Low Frontmatter date: field
  1. Score each learning: fresh / possibly stale / likely stale
  2. Report with grouped results and specific evidence for each staleness flag
  3. Act - user chooses: archive stale learnings, review one-by-one, export audit log, or done

Example output

Knowledge Freshness Audit
=========================
Scanned: 14 learnings in docs/solutions/
Date: 2026-03-08

FRESH (10)
  performance-issues/n-plus-one-briefs.md
  runtime-errors/websocket-reconnect.md
  ...

POSSIBLY STALE (3)
  database-issues/migration-ordering.md
    - Referenced file db/migrate/20250115_*.rb modified significantly (4 commits, 89 lines since 2025-11-15)
    - Module DatabaseMigrator found at new location lib/migration_runner.rb
    Action: Review and update if the solution approach has changed

LIKELY STALE (1)
  logic-errors/user-auth-token-expiry.md
    - Referenced file app/models/auth_token.rb no longer exists
    - Class AuthToken not found in codebase
    Action: Archive or rewrite

Design Decisions

Decision Choice Rationale
Flags, doesn't auto-archive Manual action required False positives could hide valid learnings
Git-based, not time-based Cross-references actual code changes Some learnings (architecture decisions) stay valid for years; time alone is too crude
Pure skill (no TypeScript) SKILL.md only Follows compound-docs pattern, zero dependency footprint
Weighted signals 5 heuristics, not binary Avoids noisy false positives from single-signal detection

Alternatives Considered

  1. Time-based expiry - Too crude. Architectural learnings stay valid for years while bug fixes go stale in weeks.
  2. Manual review cadence - Adds human toil without leveraging codebase signals already available via git.
  3. Auto-archive stale learnings - False positives could hide valid learnings. The auditor should flag, not act.

Integration Points

  • compound-docs - audits the files that compound-docs creates
  • learnings-researcher - future enhancement: surface freshness warnings when retrieving learnings
  • /ce:compound - run audit before compounding to avoid duplicating stale knowledge

Changes

  • New: plugins/compound-engineering/skills/ce-audit/SKILL.md (216 lines)
  • Updated: plugin.json, marketplace.json (version bump to 2.39.0, skill count 20 -> 21)
  • Updated: README.md (added ce-audit to skills table)
  • Updated: CHANGELOG.md (documented addition)

Test plan

  • Install plugin locally and run /ce:audit on a project with existing docs/solutions/ files
  • Verify it correctly identifies learnings referencing deleted files as "likely stale"
  • Verify it correctly identifies learnings referencing heavily modified files as "possibly stale"
  • Verify unchanged references are classified as "fresh"
  • Verify archive option moves files to docs/solutions/_archived/ with updated frontmatter
  • Verify report export creates readable audit log in docs/solutions/_audit-log/
  • Verify graceful handling when docs/solutions/ is empty or doesn't exist

🤖 Generated with Claude Code

mvanhorn and others added 2 commits March 8, 2026 12:51
Documents all missing agent-browser CLI commands for debugging workflows.
Adds 9 new command categories: eval, console/errors, network, storage,
device settings, element debugging, recording/tracing, tabs, and
advanced mouse controls.

Closes EveryInc#170

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detect stale learnings in docs/solutions/ by cross-referencing
documented solutions against the current codebase. Checks for deleted
files, renamed modules, restructured directories, and significantly
modified code. Outputs a structured freshness report with actionable
recommendations (archive, review, or export).

Addresses EveryInc#221.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@kaminskypavel
Copy link

can we get this merged? 🙏 @kieranklaassen

@vanguard9
Copy link

@kieranklaassen 👍🏻

@mvanhorn
Copy link
Contributor Author

Thanks for the interest, @kaminskypavel and @vanguard9! This is up to @kieranklaassen to review and merge - the man is the GOAT, he'll get to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants