feat: integrate PeerDB native maintenance mode into upgrade lifecycle

[Neurostep]

Description

Integrates PeerDB's native maintenance mode (PeerDB-io/peerdb#2211) into the operator's upgrade state machine. When spec.maintenance is configured on a PeerDBCluster, the operator gracefully pauses all running mirrors before upgrading components and resumes them after.

Upgrade flow with maintenance mode

Waiting → StartMaintenance → Config → InitJobs → FlowAPI → Server → UI → EndMaintenance → Complete

When spec.maintenance is not set, the upgrade flow remains unchanged (fully backwards compatible).

Changes

API (api/v1alpha1/)

• MaintenanceSpec type with image, backoffLimit, and resources fields
• spec.maintenance field on PeerDBClusterSpec
• StartMaintenance and EndMaintenance upgrade phases
• MaintenanceMode condition type and associated reason constants
• Regenerated deepcopy and CRD manifests

Resource builders (internal/resources/)

• maintenance_jobs.go: BuildStartMaintenanceJob / BuildEndMaintenanceJob — creates K8s Jobs using ghcr.io/peerdb-io/flow-maintenance:stable-{version} with start/end subcommands

Controller (internal/controller/)

• upgradePhaseMaintenanceJob — generic handler for both maintenance phases: creates Job, polls completion, deletes and retries on failure
• upgradePhaseWaiting routes to StartMaintenance when maintenance is configured
• UpgradePhaseUI routes to EndMaintenance (instead of Complete) when maintenance is configured

Documentation

• API reference: MaintenanceSpec, MaintenanceMode condition, new upgrade phases
• Architecture: updated diagram and reconciliation strategy
• Safe upgrade runbook: new Maintenance Mode section with YAML examples
• README: feature bullet

Testing

Unit tests (run locally)

go test ./...

Four new test cases verify:

1. Maintenance Job creation during upgrade
2. Advancing past StartMaintenance when Job completes
3. No maintenance phases when spec.maintenance is not set
4. Failed Job deletion/retry with Degraded condition

E2E tests (requires Kind cluster)

make test-e2e

New test creates a cluster with spec.maintenance: {}, patches version, and verifies:

• Start maintenance Job is created with correct command and ownerReference
• Upgrade status shows StartMaintenance phase

Manual verification

1. Deploy the operator to a cluster with PeerDB and external dependencies
2. Create a PeerDBCluster with spec.maintenance: {}:

   spec:
     version: "v0.36.7"
     maintenance: {}

3. Patch the version: kubectl patch peerdbcluster <name> --type merge -p '{"spec":{"version":"v0.36.8"}}'
4. Observe:
   • kubectl get job — maintenance start job created
   • kubectl get peerdbcluster <name> -o jsonpath="{.status.upgrade}" — phases progress through StartMaintenance → ... → EndMaintenance → Complete
   • kubectl get peerdbcluster <name> -o jsonpath="{.status.conditions}" — MaintenanceMode condition toggles

Related

• PeerDB maintenance mode PR: PeerDB-io/peerdb#2211 — introduces StartMaintenance / EndMaintenance Temporal workflows and the flow-maintenance container image
• PeerDB maintenance mode issue: PeerDB-io/peerdb#2174
• Maintenance image: ghcr.io/peerdb-io/flow-maintenance — runs via a dedicated Temporal task queue, ensuring compatibility across version upgrades