feat: v2.6 Cognitive Retrieval milestone (phases 39-44) by RichardHightower · Pull Request #22 · SpillwaveSolutions/agent-memory

RichardHightower · 2026-03-11T19:28:07Z

Summary

Delivers the complete v2.6 Cognitive Retrieval milestone — 6 phases, 13 plans, all implemented and verified.

Phase 39 (BM25 Hybrid Wiring): TeleportSearcher with RRF fusion for real BM25 keyword + vector hybrid search
Phase 40 (Salience + Usage Decay): Combined ranking formula (similarity * salience_factor * usage_penalty) with 50% score floor, enrichment pipeline, E2E tests
Phase 41 (Lifecycle Automation): Vector pruning CLI (prune-vectors --age-days N --dry-run), BM25 rebuild command, Bm25RebuildJob scheduler, configurable retention per level
Phase 42 (Observability RPCs): Real buffer_size in GetDedupStatus, ranking metrics (avg salience, usage decay) in GetRankingStatus, status --verbose CLI
Phase 43 (Episodic Schema & Storage): Episode/Action/ActionResult types, CF_EPISODES column family, CRUD storage ops, EpisodicConfig with value scoring
Phase 44 (Episodic gRPC & Retrieval): 4 RPCs (StartEpisode, RecordAction, CompleteEpisode, GetSimilarEpisodes), brute-force cosine similarity search, value-based retention pruning, 6 E2E tests

Key Changes

HybridSearchHandler uses real BM25 via TeleportSearcher (was hardcoded unavailable)
Combined ranking: original * salience_factor * usage_penalty with configurable score floor
Lifecycle config: vector pruning enabled by default, BM25 opt-in, per-level retention
Admin CLI: prune-vectors and rebuild-bm25 commands
Verbose status: memory-daemon status --verbose shows dedup, ranking, vector health
Episodic memory: full Episode lifecycle with value scoring, similarity search, and retention pruning
Proto fields >200 for Phase 40+ additions (access_count, last_accessed_ms, ranking metrics, episode RPCs)

Test plan

cargo fmt --all -- --check passes
cargo clippy --workspace --all-targets --all-features -- -D warnings clean
cargo test --workspace --all-features — 47 suites, 0 failures
E2E ranking tests verify salience enrichment and usage decay ordering
E2E episodic tests cover create, get, list, search, and value-based retention
E2E hybrid search tests verify BM25 fallback and mode switching
CI pipeline confirms all checks green

🤖 Generated with Claude Code

…mory

Synthesized research across STACK, FEATURES, ARCHITECTURE, and PITFALLS for Agent Memory v2.6 (Episodic Memory, Ranking Quality, Lifecycle & Observability). SUMMARY.md consolidates findings into phase implications (Phases 39-42) with confidence assessment and roadmap flags. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Phase 39: BM25 Hybrid Wiring (2 plans) Phase 40: Salience + Usage Decay (3 plans) Phase 41: Lifecycle Automation (2 plans) Phase 42: Observability RPCs (2 plans) Phase 43: Episodic Schema & Storage (1 plan) Phase 44: Episodic gRPC & Retrieval (3 plans) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Episode struct with ULID id, task, plan, actions, status, scoring - Action struct with type, input, result, timestamp - ActionResult enum: Success, Failure, Pending - EpisodeStatus enum: InProgress, Completed, Failed - Value scoring formula for retrieval prioritization - Serialize/Deserialize with serde(default) for backward compat Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Add CF_EPISODES constant and include in ALL_CF_NAMES - Add ColumnFamilyDescriptor with default Options in build_cf_descriptors - Export CF_EPISODES from crate root Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- store_episode, get_episode, list_episodes, update_episode, delete_episode - list_episodes returns newest first via reverse ULID iteration - Uses CF_EPISODES column family with JSON serialization - 7 unit tests for round-trip, CRUD, ordering, and complex actions - Made Storage.db field pub(crate) for cross-module CF access Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- EpisodicConfig: enabled, value_threshold, midpoint_target, max_episodes - Wired into Settings with [episodic] TOML section - Disabled by default (explicit opt-in like dedup) - Validation for all fields - Backward compatible with serde(default) for pre-phase-43 configs - 6 unit tests for defaults, validation, serialization, backward compat Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- SUMMARY.md with all commits, deviations, decisions - STATE.md updated with progress (1/13 plans, 8%) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Bridge Storage salience/access_count data into search result metadata so the combined ranking formula (salience boost + usage decay) takes effect. Add enrich_with_salience() that lookups TocNode/Grip per result. Update StaleFilter with improved kind exemptions and configurable decay. Add usage tracking fields to TocNode (access_count, last_accessed_ms). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Add access_count and last_accessed_ms to TocNode (type + proto) - Add salience and usage config sections to Settings - Implement combined ranking formula: similarity * salience_factor * usage_penalty - Add 50% score floor to prevent collapse - Enrich search results with salience/usage metadata from Storage lookups - Wire combined ranking into retrieval pipeline after stale filter - Add ranking E2E tests (salience order, usage decay, score floor) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add 3 E2E tests exercising the full route_query pipeline: - Salience enrichment from Storage affects ranking order (RANK-09) - Access count enrichment flows through pipeline (RANK-10) - Ranking composes with StaleFilter (constraint exemption + salience) Plus 4 unit-level tests for the ranking formula. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Re-apply Phase 39 changes that were accidentally reverted: - TeleportSearcher field in HybridSearchHandler - Real bm25_available() and bm25_search() implementations - Constructor wiring in with_all_services/with_all_services_and_topics - E2E hybrid search test Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Add LifecycleConfig with VectorLifecycleSettings and Bm25LifecycleSettings - Add PruneVectors admin CLI command with --age-days, --dry-run flags - Add RebuildBm25 admin CLI command with --min-level filter - Implement handle_prune_vectors using VectorIndexPipeline per-level pruning - Implement handle_rebuild_bm25 using SearchIndexer level-based filtering - Create Bm25RebuildJob scheduler job with cron, cancellation, rebuild callback - Register both prune jobs on daemon startup via register_prune_jobs - Vector pruning enabled by default; BM25 lifecycle disabled (opt-in) - Default retention: segment=30d, grip=30d, day=365d, week=1825d Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… status - Fix GetDedupStatus to return real buffer_size via NoveltyChecker::buffer_len() - Add ranking metrics to GetRankingStatusResponse proto (field numbers >200): avg_salience_score, high_salience_count, total_access_count, avg_usage_decay - Compute ranking metrics from recent day-level TOC nodes (30-day window) - Wire live novelty counters into GetRankingStatus response - Add get_dedup_status and get_ranking_status methods to MemoryClient - Add verbose status CLI: `memory-daemon status --verbose` shows dedup, ranking, vector - Add SearchIndexer::rebuild_with_filter for level-based BM25 rebuild Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Resolve conflicts in config.rs and lib.rs to combine Phase 41 lifecycle settings and Phase 43 episodic config. Both feature sets coexist in Settings struct with serde defaults. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ests - Add proto definitions: StartEpisode, RecordAction, CompleteEpisode, GetSimilarEpisodes RPCs - Add EpisodeAction, EpisodeSummary, EpisodeStatusProto proto messages - Implement EpisodeHandler with Arc<Storage> and EpisodicConfig - Wire episode RPCs into MemoryServiceImpl via set_episode_handler() - Implement value-based retention pruning (lowest value_score pruned at max_episodes) - Implement brute-force cosine similarity search for episodes - Generate episode embeddings on completion (task + lessons text) - Add 6 E2E tests: lifecycle, value-based pruning, disabled config, error cases - Add 15 unit tests in EpisodeHandler Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

All 6 phases executed: - Phase 39: BM25 hybrid wiring - Phase 40: Salience scoring + usage decay - Phase 41: Lifecycle automation (vector prune + BM25 rebuild) - Phase 42: Observability RPCs (dedup buffer, ranking metrics) - Phase 43: Episodic schema & storage - Phase 44: Episodic gRPC, similarity search, value retention Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

RichardHightower and others added 18 commits March 11, 2026 01:42

docs: start milestone v2.6 Retrieval Quality, Lifecycle & Episodic Me…

9baa8ad

…mory

docs: define milestone v2.6 requirements

982ecd6

docs: create milestone v2.6 roadmap (6 phases)

5580747

feat(43-01): add CF_EPISODES column family for episodic memory

0421c2e

- Add CF_EPISODES constant and include in ALL_CF_NAMES - Add ColumnFamilyDescriptor with default Options in build_cf_descriptors - Export CF_EPISODES from crate root Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

chore(43-01): apply cargo fmt formatting fixes

f7608d3

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge branch 'main' into feature/phase-43-episodic-schema-storage

d40c677

docs(43-01): complete Episode Schema, Storage, and Column Family plan

01451b6

- SUMMARY.md with all commits, deviations, decisions - STATE.md updated with progress (1/13 plans, 8%) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

RichardHightower temporarily deployed to e2e-cli March 11, 2026 19:28 — with GitHub Actions Inactive

RichardHightower and others added 2 commits March 11, 2026 21:53

RichardHightower and others added 2 commits March 11, 2026 22:18

Merge phase-44 episodic gRPC into feature branch

e834b7b

RichardHightower temporarily deployed to e2e-cli March 12, 2026 05:30 — with GitHub Actions Inactive

RichardHightower had a problem deploying to e2e-cli March 12, 2026 05:30 — with GitHub Actions Failure

RichardHightower temporarily deployed to e2e-cli March 12, 2026 05:30 — with GitHub Actions Inactive

RichardHightower changed the title ~~feat: v2.6 phases 39-42 - retrieval quality, lifecycle, observability~~ feat: v2.6 Cognitive Retrieval milestone (phases 39-44) Mar 12, 2026

fix: escape Arc<Storage> in rustdoc comment to fix doc build

bf11c84

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

RichardHightower temporarily deployed to e2e-cli March 12, 2026 12:56 — with GitHub Actions Inactive

RichardHightower merged commit be40813 into main Mar 12, 2026
20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: v2.6 Cognitive Retrieval milestone (phases 39-44)#22

feat: v2.6 Cognitive Retrieval milestone (phases 39-44)#22
RichardHightower merged 23 commits intomainfrom
feature/phase-40-salience-usage-decay

RichardHightower commented Mar 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

RichardHightower commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

Test plan

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

RichardHightower commented Mar 11, 2026 •

edited

Loading