Skip to content

Feat(dbt_cli): Set proper plan flags and also allow --empty and --environment#5226

Merged
erindru merged 7 commits intomainfrom
erin/dbt-run-flags
Aug 29, 2025
Merged

Feat(dbt_cli): Set proper plan flags and also allow --empty and --environment#5226
erindru merged 7 commits intomainfrom
erin/dbt-run-flags

Conversation

@erindru
Copy link
Collaborator

@erindru erindru commented Aug 26, 2025

This PR:

  • adds the --empty dbt option which maps to SQLMesh --skip-backfill (creates tables but doesnt write data to them).
  • adds the --environment option to allow creating development VDE's. This is not a native dbt option but is a value-add of the SQLMesh engine.

In addition, this PR tightens up the plan flags to make SQLMesh more closely mimic dbt behaviour:

  • All plans have --run and --ignore_cron set - except dev plans with no --select.
    This mimics the dbt behaviour of "always run up until right now" while falling back to the default SQLMesh behaviour for dev plans where the user has no interest in a particular set of models
  • Dev plans set include_unmodifed=False to create dev environments with just the changes, create_from=prod,always_recreate_environment=True to ensure that dev environments are always based off the latest prod at the time the command is run and enable_preview=None to use default behaviour for dev previews.

Caveats:

  • enable_preview=True is deliberately set even though this will cause changed dbt incremental models to be fully refreshed in dev environments. The assumption is that if the user is opting in to VDE, and this is a problem, then they may also consider opting in to SQLMesh incremental models
  • --empty in dbt replaces existing tables with empty ones, but --skip-backfill just skips the backfill step and wont touch a table if it didnt already need to be backfilled. Therefore, this is a partial implementation of --empty until we have everything in place to enable --full-refresh semantics

@erindru erindru force-pushed the erin/dbt-run-flags branch 2 times, most recently from defb7fd to 0b21555 Compare August 26, 2025 21:02
help="If specified, dbt will drop incremental models and fully-recalculate the incremental table from the model definition.",
)
@click.option(
"--environment",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we add --env shortcut? or maybe make it an argument instead of an option? like we do for a plan? feel intuitive. Plus it doesn't look like dbt run has any arguments.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would feel intuitive for a sqlmesh user, but people used to running dbt run are used to specifying a boatload of options, right?

dbt run --help reveals that the original command has no arguments, so the established practice appears to be to keep adding options

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did add --env as an alias

# to prevent the following error:
# > ConfigError: When targeting the production environment either the backfill should not be skipped or
# > the lack of data gaps should be enforced (--no-gaps flag).
options["no_gaps"] = True
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if we should even do this.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldnt get --skip-backfill to work without it in my example project?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After internal discussion, i've changed the ConfigError to a warning and stopped setting no_gaps to get around it.

@erindru erindru force-pushed the erin/dbt-run-flags branch 2 times, most recently from 71c9a47 to 07e8c03 Compare August 27, 2025 21:19
@erindru erindru force-pushed the erin/dbt-run-flags branch from 07e8c03 to 27d1f3d Compare August 28, 2025 04:50
@erindru erindru merged commit 6d36a11 into main Aug 29, 2025
35 of 36 checks passed
@erindru erindru deleted the erin/dbt-run-flags branch August 29, 2025 00:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants