Skip to content

feat: Airbyte snapshot pull + seed-from-snapshot workflow#30

Closed
zozo123 wants to merge 1 commit intomainfrom
codex/airbyte-snapshot-single-pr
Closed

feat: Airbyte snapshot pull + seed-from-snapshot workflow#30
zozo123 wants to merge 1 commit intomainfrom
codex/airbyte-snapshot-single-pr

Conversation

@zozo123
Copy link
Contributor

@zozo123 zozo123 commented Feb 18, 2026

Summary

This PR adds an end-to-end snapshot ingestion path in one place:

  • New doubleagent snapshot command family:
    • pull, list, inspect, delete
  • Connector schema in service.yaml (connector block) with Airbyte support
  • Airbyte pull helper at services/_lib/snapshot_pull/
    • pyairbyte backend
    • optional smart relational filtering
    • deterministic PII redaction
    • snapshot persistence (manifest.json, seed.json, per-resource JSON)
  • doubleagent seed <service> --snapshot <profile>
    • seeds directly from ~/.doubleagent/snapshots/<service>/<profile>/seed.json
  • GitHub service now includes an example Airbyte connector config
  • Docs updated in README and contributing guide

Rationale

Single PR is intentional here because these parts are tightly coupled and only provide user value together:

  1. snapshot pull without seed integration leaves a dead-end artifact flow.
  2. seed --snapshot without connector pull has no built-in source of truth.
  3. Connector schema and helper runtime must land with the CLI surface, otherwise command UX is broken.

This keeps the model simple:

Airbyte pull -> local snapshot profile -> existing /_doubleagent/seed path

No new runtime state model, no namespace/COW/bootstrap mechanics, and no change to the required service interface.

Behavior notes

  • Snapshot pulls fail fast when required env vars are missing (connector.required_env).
  • Redaction is on by default; --no-redact is explicit.
  • Strict compliance mode (DOUBLEAGENT_COMPLIANCE_MODE=strict) blocks snapshot pulls.
  • Airbyte backend currently supports pyairbyte only.

Validation

  • cargo fmt
  • cargo check
  • cargo test -p doubleagent-core
  • cargo test -p doubleagent
  • cargo run -p doubleagent -- snapshot --help
  • cargo run -p doubleagent -- seed --help
  • python3 -m snapshot_pull --help (from services/_lib)

@zozo123 zozo123 requested review from AdamGold, Copilot, rotemtam and tomerezer and removed request for Copilot and tomerezer February 19, 2026 21:55
zozo123 added a commit that referenced this pull request Feb 23, 2026
Resolve conflicts between the snapshot and fixture features:
- --snapshot flag seeds from a snapshot profile
- --fixture flag resolves fixtures from service fixtures/ dir
- Mutual exclusion: only one of --snapshot, --fixture, or file path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@zozo123
Copy link
Contributor Author

zozo123 commented Feb 23, 2026

Superseded by #33 (consolidated PR)

@zozo123 zozo123 closed this Feb 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant