Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions docs/concepts/models/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -507,11 +507,18 @@ Some properties are only available in specific model kinds - see the [model conf
: Set this to true to indicate that all changes to this model should be [forward-only](../plans.md#forward-only-plans).

### on_destructive_change
: What should happen when a change to a [forward-only model](../../guides/incremental_time.md#forward-only-models) or incremental model in a [forward-only plan](../plans.md#forward-only-plans) causes a destructive modification to the table schema (i.e., requires dropping an existing column).
: What should happen when a change to a [forward-only model](../../guides/incremental_time.md#forward-only-models) or incremental model in a [forward-only plan](../plans.md#forward-only-plans) causes a destructive modification to the table schema (i.e., requires dropping an existing column or modifying column constraints in ways that could cause data loss).

SQLMesh checks for destructive changes at plan time based on the model definition and run time based on the model's underlying physical tables.

Must be one of the following values: `allow`, `warn`, or `error` (default).
Must be one of the following values: `allow`, `warn`, `error` (default), or `ignore`.

### on_additive_change
: What should happen when a change to a [forward-only model](../../guides/incremental_time.md#forward-only-models) or incremental model in a [forward-only plan](../plans.md#forward-only-plans) causes an additive modification to the table schema (i.e., adding new columns, modifying column data types in compatible ways, ect.).

SQLMesh checks for additive changes at plan time based on the model definition and run time based on the model's underlying physical tables.

Must be one of the following values: `allow` (default), `warn`, `error`, or `ignore`.

### disable_restatement
: Set this to true to indicate that [data restatement](../plans.md#restatement-plans) is disabled for this model.
Expand Down
20 changes: 18 additions & 2 deletions docs/concepts/plans.md
Original file line number Diff line number Diff line change
Expand Up @@ -354,9 +354,25 @@ Some model changes destroy existing data in a table. SQLMesh automatically detec

Forward-only plans treats all of the plan's model changes as forward-only. In these plans, SQLMesh will check all modified incremental models for destructive schema changes, not just forward-only models.

SQLMesh determines what to do for each model based on this setting hierarchy: the [model's `on_destructive_change` value](../guides/incremental_time.md#destructive-changes) (if present), the `on_destructive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`.
SQLMesh determines what to do for each model based on this setting hierarchy:

If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's `--allow-destructive-model` selector to specify which models. Learn more about model selectors [here](../guides/model_selection.md).
- **For destructive changes**: the [model's `on_destructive_change` value](../guides/incremental_time.md#schema-changes) (if present), the `on_destructive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`
- **For additive changes**: the [model's `on_additive_change` value](../guides/incremental_time.md#schema-changes) (if present), the `on_additive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `allow`

If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's `--allow-destructive-model` selector to specify which models.
Similarly, if you want to temporarily allow additive changes to models configured with `on_additive_change=error`, use the `--allow-additive-model` selector.

For example, to allow additive changes to all models in the `analytics` schema:
```bash
sqlmesh plan --forward-only --allow-additive-model "analytics.*"
```

Or to allow additive changes to multiple specific models:
```bash
sqlmesh plan --forward-only --allow-additive-model "sales.revenue_model" --allow-additive-model "marketing.campaign_model"
```

Learn more about model selectors [here](../guides/model_selection.md).

### Effective date
Changes that are part of the forward-only plan can also be applied retroactively to the production environment by specifying the effective date:
Expand Down
133 changes: 123 additions & 10 deletions docs/guides/incremental_time.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,19 +159,47 @@ WHERE

Alternatively, all the changes contained in a *specific plan* can be classified as forward-only with a flag: `sqlmesh plan --forward-only`. A subsequent plan that did not include the forward-only flag would fully refresh the model's physical table. Learn more about forward-only plans [here](../concepts/plans.md#forward-only-plans).

### Destructive changes
### Schema changes

Some model changes destroy existing data in a table. Dropping a column from the model is the most direct cause, but changing a column's data type (such as casting a column from a `STRING` to `INTEGER`) can also require a drop. (Whether or not a specific change requires dropping a column may differ across SQL engines.)
When SQLMesh processes forward-only changes to incremental models, it compares the model's new schema with the existing physical table schema to detect potential data loss or compatibility issues. SQLMesh categorizes schema changes into two types:

Forward-only models are used to retain existing data. Before executing forward-only changes to incremental models, SQLMesh performs a check to determine if existing data will be destroyed.
#### Destructive changes
Some model changes destroy existing data in a table. Examples include:

The check is performed at plan time based on the model definition. SQLMesh may not be able to resolve all of a model's column data types and complete the check, so the check is performed again at run time based on the physical tables underlying the model.
- **Dropping a column** from the model
- **Renaming a column** without proper aliasing
- **Modifying column constraints** in ways that could cause data loss

Whether a specific change is destructive may differ across SQL engines based on their schema evolution capabilities.

#### Additive changes
Additive changes are any changes that aren't categorized as destructive. A simple examples would be adding a column to a table.

SQLMesh performs schema change detection at plan time based on the model definition. If SQLMesh cannot resolve all of a model's column data types at plan time, the check is performed again at run time based on the physical tables underlying the model.

#### Changes to forward-only models

A model's `on_destructive_change` [configuration setting](../reference/model_configuration.md#incremental-models) determines what happens when SQLMesh detects a destructive change.
SQLMesh provides two configuration settings to control how schema changes are handled:

- **`on_destructive_change`** - Controls behavior for destructive schema changes
- **`on_additive_change`** - Controls behavior for additive schema changes

##### Configuration options

Both properties support four values:

- **`error`** (default for `on_destructive_change`): Stop execution and raise an error
- **`warn`**: Log a warning but proceed with the change
- **`allow`** (default for `on_additive_change`): Silently proceed with the change
- **`ignore`**: Skip the schema change check entirely for this change type

!!! warning "Ignore is Dangerous"

By default, SQLMesh will error so no data is lost. You can set `on_destructive_change` to `warn` or `allow` in the model's `MODEL` block to allow destructive changes.
`ignore` is dangerous since it can result in error or data loss. It likely should never be used but could be useful as an "escape-hatch" or a way to workaround unexpected behavior.

##### Destructive change handling

The `on_destructive_change` [configuration setting](../reference/model_configuration.md#incremental-models) determines what happens when SQLMesh detects a destructive change. By default, SQLMesh will error so no data is lost.

This example configures a model to silently `allow` destructive changes:

Expand All @@ -186,12 +214,97 @@ MODEL (
);
```

A default `on_destructive_change` value can be set for all incremental models that do not specify it themselves in the [model defaults configuration](../reference/model_configuration.md#model-defaults).
##### Additive change handling

The `on_additive_change` configuration setting determines what happens when SQLMesh detects an additive change like adding new columns. By default, SQLMesh allows these changes since they don't destroy existing data.

This example configures a model to raise an error for additive changes (useful for strict schema control):

``` sql linenums="1"
MODEL (
name sqlmesh_example.new_model,
kind INCREMENTAL_BY_TIME_RANGE (
time_column model_time_column,
forward_only true,
on_additive_change error
),
);
```

##### Combining both settings

You can configure both settings together to have fine-grained control over schema evolution:

``` sql linenums="1"
MODEL (
name sqlmesh_example.new_model,
kind INCREMENTAL_BY_TIME_RANGE (
time_column model_time_column,
forward_only true,
on_destructive_change warn, -- Warn but allow destructive changes
on_additive_change allow -- Silently allow new columns
),
);
```

!!! warning "Unusual combinations"

SQLMesh will warn about unusual combinations such as allowing destructive changes while erroring on additive changes, since this indicates a potentially inconsistent schema change policy.

##### Model defaults

Default values for both `on_destructive_change` and `on_additive_change` can be set for all incremental models in the [model defaults configuration](../reference/model_configuration.md#model-defaults).

##### Common use cases

Here are some common patterns for configuring schema change handling:

**Strict schema control** - Prevent any schema changes:
```sql linenums="1"
MODEL (
name sqlmesh_example.strict_model,
kind INCREMENTAL_BY_TIME_RANGE (
time_column event_date,
forward_only true,
on_destructive_change error, -- Block destructive changes
on_additive_change error -- Block even new columns
),
);
```

**Permissive development model** - Allow all schema changes:
```sql linenums="1"
MODEL (
name sqlmesh_example.dev_model,
kind INCREMENTAL_BY_TIME_RANGE (
time_column event_date,
forward_only true,
on_destructive_change allow, -- Allow dropping columns
on_additive_change allow -- Allow new columns
),
);
```

**Production safety** - Allow safe changes, warn about risky ones:
```sql linenums="1"
MODEL (
name sqlmesh_example.production_model,
kind INCREMENTAL_BY_TIME_RANGE (
time_column event_date,
forward_only true,
on_destructive_change warn, -- Warn about destructive changes
on_additive_change allow -- Silently allow new columns
),
);
```

#### Changes in forward-only plans

The SQLMesh `plan` [`--forward-only` option](../concepts/plans.md#forward-only-plans) treats all the plan's model changes as forward-only. When this option is specified, SQLMesh will check all modified incremental models for destructive schema changes, not just models configured with `forward_only true`.
The SQLMesh `plan` [`--forward-only` option](../concepts/plans.md#forward-only-plans) treats all the plan's model changes as forward-only. When this option is specified, SQLMesh will check all modified incremental models for both destructive and additive schema changes, not just models configured with `forward_only true`.

SQLMesh determines what to do for each model based on this setting hierarchy:

SQLMesh determines what to do for each model based on this setting hierarchy: the model's `on_destructive_change` value (if present), the `on_destructive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`.
- **For destructive changes**: the model's `on_destructive_change` value (if present), the `on_destructive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`
- **For additive changes**: the model's `on_additive_change` value (if present), the `on_additive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `allow`

If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's [`--allow-destructive-model` selector](../concepts/plans.md#destructive-changes) to specify which models. Learn more about model selectors [here](../guides/model_selection.md).
If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's [`--allow-destructive-model` selector](../concepts/plans.md#destructive-changes) to specify which models. Similarly, if you want to temporarily allow additive changes to models configured with `on_additive_change=error`, use the [`--allow-additive-model` selector](../concepts/plans.md#destructive-changes). Learn more about model selectors [here](../guides/model_selection.md).
2 changes: 1 addition & 1 deletion docs/guides/model_selection.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This guide describes how to select specific models to include in a SQLMesh plan, which can be useful when modifying a subset of the models in a SQLMesh project.

Note: the selector syntax described below is also used for the SQLMesh `plan` [`--allow-destructive-model` selector](../concepts/plans.md#destructive-changes) and for the `table_diff` command to [diff a selection of models](./tablediff.md#diffing-multiple-models-across-environments).
Note: the selector syntax described below is also used for the SQLMesh `plan` [`--allow-destructive-model` and `--allow-additive-model` selectors](../concepts/plans.md#destructive-changes) and for the `table_diff` command to [diff a selection of models](./tablediff.md#diffing-multiple-models-across-environments).

## Background

Expand Down
18 changes: 9 additions & 9 deletions docs/integrations/dbt.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,18 +211,18 @@ Similarly, the [allow_partials](../concepts/models/overview.md#allow_partials) p

#### on_schema_change

SQLMesh automatically detects destructive schema changes to [forward-only incremental models](../guides/incremental_time.md#forward-only-models) and to all incremental models in [forward-only plans](../concepts/plans.md#destructive-changes).
SQLMesh automatically detects both destructive and additive schema changes to [forward-only incremental models](../guides/incremental_time.md#forward-only-models) and to all incremental models in [forward-only plans](../concepts/plans.md#destructive-changes).

A model's [`on_destructive_change` setting](../guides/incremental_time.md#destructive-changes) determines whether it errors (default), warns, or silently allows the changes. SQLMesh always allows non-destructive forward-only schema changes, such as adding or casting a column in place.
A model's [`on_destructive_change` and `on_additive_change` settings](../guides/incremental_time.md#schema-changes) determine whether it errors, warns, silently allows, or ignores the changes. SQLMesh provides fine-grained control over both destructive changes (like dropping columns) and additive changes (like adding new columns).

`on_schema_change` configuration values are mapped to these SQLMesh `on_destructive_change` values:
`on_schema_change` configuration values are mapped to these SQLMesh settings:

| `on_schema_change` | SQLMesh `on_destructive_change` |
| ------------------ | ------------------------------- |
| ignore | warn |
| append_new_columns | warn |
| sync_all_columns | allow |
| fail | error |
| `on_schema_change` | SQLMesh `on_destructive_change` | SQLMesh `on_additive_change` |
|--------------------|---------------------------------|------------------------------|
| ignore | ignore | ignore |
| fail | error | error |
| append_new_columns | ignore | allow |
| sync_all_columns | allow | allow |


## Snapshot support
Expand Down
2 changes: 2 additions & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -367,6 +367,8 @@ Options:
--forward-only Create a plan for forward-only changes.
--allow-destructive-model TEXT Allow destructive forward-only changes to
models whose names match the expression.
--allow-additive-model TEXT Allow additive forward-only changes to
models whose names match the expression.
--effective-from TEXT The effective date from which to apply
forward-only changes on production.
--no-prompts Disable interactive prompts for the backfill
Expand Down
Loading