Store sync plan in bucket storage

[simolus3]

Sync configurations derived from the new compiler have a stable serialized format, meaning that we can store them as JSON data in bucket storage and retrieve it from there instead of recompiling every time the replicator starts. This allows us to evolve the compiler without breaking existing sync plans (which would have been the case before since the parameter format needs to match exactly between replication and the querier).

When deploying a sync config from YAML, it will get parsed and passed through updateSyncRulesFromParsed which attaches a sync plan if possible. When reading sync configurations (in PersistedSyncRulesContent.parsed(), we now take the sync plan into account and prefer to deserialize instead of re-compiling).

Testing

I've tested this locally with the both mongo and Postgres storage implementations. First, I've deployed this sync config:

streams:
  lists:
    query: SELECT * FROM lists
    auto_subscribe: true
  todos:
    query: SELECT * FROM todos WHERE list_id = subscription.parameter('list')

config:
  edition: 2
  sync_config_compiler: true

After deploying that, I've stopped the sync service and applied this patch to prevent the service from properly compiling new streams:

diff --git a/packages/sync-rules/src/from_yaml.ts b/packages/sync-rules/src/from_yaml.ts
index 124d341e..5252519a 100644
--- a/packages/sync-rules/src/from_yaml.ts
+++ b/packages/sync-rules/src/from_yaml.ts
@@ -222,7 +222,8 @@ export class SyncConfigFromYaml {
       streamCompiler.finish();
     }
 
-    const config = new PrecompiledSyncConfig(compiler.output.toSyncPlan(), {
+    const emptyCompiler = new SyncStreamsCompiler(this.options);
+    const config = new PrecompiledSyncConfig(emptyCompiler.output.toSyncPlan(), {
       defaultSchema: this.options.defaultSchema,
       engine: javaScriptExpressionEngine(compatibility),
       sourceText: this.yaml

After restarting the sync service, I've connected with the supabase todolist sync streams client to ensure data is there. If the service had re-compiled sync streams for the querier, I would not have seen any data since no streams would be available with that patch.

[changeset-bot]

🦋 Changeset detected

Latest commit: d92e125

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 18 packages

Name	Type
@powersync/service-sync-rules	Patch
@powersync/service-module-postgres-storage	Minor
@powersync/service-module-mongodb-storage	Minor
@powersync/service-core	Minor
@powersync/service-jpgwire	Patch
@powersync/service-core-tests	Patch
@powersync/lib-services-framework	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-schema	Minor
@powersync/service-image	Minor
@powersync/service-module-core	Patch
test-client	Patch
@powersync/lib-service-postgres	Patch
@powersync/service-rsocket-router	Patch
@powersync/lib-service-mongodb	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR