Feat(dbt_cli): Set proper plan flags and also allow --empty and --environment#5226
Feat(dbt_cli): Set proper plan flags and also allow --empty and --environment#5226
Conversation
defb7fd to
0b21555
Compare
| help="If specified, dbt will drop incremental models and fully-recalculate the incremental table from the model definition.", | ||
| ) | ||
| @click.option( | ||
| "--environment", |
There was a problem hiding this comment.
should we add --env shortcut? or maybe make it an argument instead of an option? like we do for a plan? feel intuitive. Plus it doesn't look like dbt run has any arguments.
There was a problem hiding this comment.
I think it would feel intuitive for a sqlmesh user, but people used to running dbt run are used to specifying a boatload of options, right?
dbt run --help reveals that the original command has no arguments, so the established practice appears to be to keep adding options
There was a problem hiding this comment.
I did add --env as an alias
sqlmesh_dbt/operations.py
Outdated
| # to prevent the following error: | ||
| # > ConfigError: When targeting the production environment either the backfill should not be skipped or | ||
| # > the lack of data gaps should be enforced (--no-gaps flag). | ||
| options["no_gaps"] = True |
There was a problem hiding this comment.
I don't know if we should even do this.
There was a problem hiding this comment.
I couldnt get --skip-backfill to work without it in my example project?
There was a problem hiding this comment.
After internal discussion, i've changed the ConfigError to a warning and stopped setting no_gaps to get around it.
71c9a47 to
07e8c03
Compare
07e8c03 to
27d1f3d
Compare
This PR:
--emptydbt option which maps to SQLMesh--skip-backfill(creates tables but doesnt write data to them).--environmentoption to allow creating development VDE's. This is not a native dbt option but is a value-add of the SQLMesh engine.In addition, this PR tightens up the plan flags to make SQLMesh more closely mimic dbt behaviour:
--runand--ignore_cronset - except dev plans with no--select.This mimics the dbt behaviour of "always run up until right now" while falling back to the default SQLMesh behaviour for dev plans where the user has no interest in a particular set of models
include_unmodifed=Falseto create dev environments with just the changes,create_from=prod,always_recreate_environment=Trueto ensure that dev environments are always based off the latest prod at the time the command is run andenable_preview=Noneto use default behaviour for dev previews.Caveats:
enable_preview=Trueis deliberately set even though this will cause changed dbt incremental models to be fully refreshed in dev environments. The assumption is that if the user is opting in to VDE, and this is a problem, then they may also consider opting in to SQLMesh incremental models--emptyin dbt replaces existing tables with empty ones, but--skip-backfilljust skips the backfill step and wont touch a table if it didnt already need to be backfilled. Therefore, this is a partial implementation of--emptyuntil we have everything in place to enable--full-refreshsemantics