Sync streams evaluator

[simolus3]

This builds on #451: While that PR adds a tool compiling sync streams from SQL to a sync plan, this adds an evaluator implementing the BucketSource interface over sync plans.

The foundational layer to evaluate rows, parameters and queries is ScalarExpressionEngine (in scalar_expression_engine.ts). This interface evaluates SQL queries of the form SELECT $outputs WHERE $filter, where both outputs and filter are deterministic and scalar expressions. There are two implementations:

1. One based on our existing tools implementing SQL expression in JavaScript.
2. A new one based on turning expression trees into SQL text that is then evaluated with node:sqlite.

Due to some behavioral differences between SQLite and our implementation, the SQL engine is not a drop-in replacement. But having it around helps validate that streams work with both, so we could introduce an option to let users migrate in the future.

Bucket data sources and parameter index lookup creators are implemented in a fairly straightforward way: In both cases, we have scalar filter expressions as well as output columns and parameter keys. These are all evaluated and used to create bucket parameters or lookups, respectively.

The most complex part of evaluating sync streams is finding parameters for connections. In sync plans, parameter instantiations are represented as ParameterValues, which can be:

1. Derived from request parameters (e.g. the scalar expression in WHERE id = auth.user_id()).
2. Derived from a lookup result, where a lookup is either:
   a. A table-valued function using request parameters as input (e.g. WHERE id IN auth.param('orgs')).
   b. An index parameter lookup (e.g. WHERE id IN (SELECT org_id FROM orgs WHERE ...)).

Lookups are further categorized in stages, where lookups from later stages are allowed to use lookups from earlier stages as inputs. Outputs from a lookup may be used by multiple later lookups or ParameterValues, which means that the querier is a DAG structure.
This structure is evaluated in RequestParameterEvaluators of parameter_evaluator.ts. The main idea is that nodes in the lookup graph are evaluated and replaced with their instantiation (essentially a form of caching). We perform two evaluation runs:

1. First, we evaluate nodes only depending on current request/token/subscripton parameters. If no nodes in the graph reference parameter lookups, the graph is fully resolved at this point and we can return all instantiations as a static querier.
2. Otherwise, we return a dynamic querier. We can clone the partial static instantiation since parameters don't change within a connection. Then, to resolve buckets dynamically, we traverse lookup stages and replace lookups with their resolved instantiation until we've completed the graph and end up with an instantiation.

Some notes:

1. Step one could be split into two, one to evaluate parameters only depending on global request and token parameters and a second one to instantiate with subscription parameters. That might reduce some work for streams with many subscriptions, but I don't expect that to have a large impact.
2. We currently can't cache anything from step 2, any change in parameter lookups requires re-running everything. In the future, we might want to consider a querier interface that dispatches which lookups specifically have changed when queryDynamicBucketDescriptions is called. This would allow us to do fine-grained invalidation and potentially improve performance.

[changeset-bot]

⚠️ No Changeset found

Latest commit: f793fab

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[rkistner]

Some comments on supported functions