You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/concepts/models/overview.md
+9-2Lines changed: 9 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -507,11 +507,18 @@ Some properties are only available in specific model kinds - see the [model conf
507
507
: Set this to true to indicate that all changes to this model should be [forward-only](../plans.md#forward-only-plans).
508
508
509
509
### on_destructive_change
510
-
: What should happen when a change to a [forward-only model](../../guides/incremental_time.md#forward-only-models) or incremental model in a [forward-only plan](../plans.md#forward-only-plans) causes a destructive modification to the table schema (i.e., requires dropping an existing column).
510
+
: What should happen when a change to a [forward-only model](../../guides/incremental_time.md#forward-only-models) or incremental model in a [forward-only plan](../plans.md#forward-only-plans) causes a destructive modification to the table schema (i.e., requires dropping an existing column or modifying column constraints in ways that could cause data loss).
511
511
512
512
SQLMesh checks for destructive changes at plan time based on the model definition and run time based on the model's underlying physical tables.
513
513
514
-
Must be one of the following values: `allow`, `warn`, or `error` (default).
514
+
Must be one of the following values: `allow`, `warn`, `error` (default), or `ignore`.
515
+
516
+
### on_additive_change
517
+
: What should happen when a change to a [forward-only model](../../guides/incremental_time.md#forward-only-models) or incremental model in a [forward-only plan](../plans.md#forward-only-plans) causes an additive modification to the table schema (i.e., adding new columns, modifying column data types in compatible ways, ect.).
518
+
519
+
SQLMesh checks for additive changes at plan time based on the model definition and run time based on the model's underlying physical tables.
520
+
521
+
Must be one of the following values: `allow` (default), `warn`, `error`, or `ignore`.
515
522
516
523
### disable_restatement
517
524
: Set this to true to indicate that [data restatement](../plans.md#restatement-plans) is disabled for this model.
Copy file name to clipboardExpand all lines: docs/concepts/plans.md
+18-2Lines changed: 18 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -354,9 +354,25 @@ Some model changes destroy existing data in a table. SQLMesh automatically detec
354
354
355
355
Forward-only plans treats all of the plan's model changes as forward-only. In these plans, SQLMesh will check all modified incremental models for destructive schema changes, not just forward-only models.
356
356
357
-
SQLMesh determines what to dofor each model based on this setting hierarchy: the [model's `on_destructive_change` value](../guides/incremental_time.md#destructive-changes) (if present), the `on_destructive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`.
357
+
SQLMesh determines what to dofor each model based on this setting hierarchy:
358
358
359
-
If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's `--allow-destructive-model` selector to specify which models. Learn more about model selectors [here](../guides/model_selection.md).
359
+
- **For destructive changes**: the [model's `on_destructive_change` value](../guides/incremental_time.md#schema-changes) (if present), the `on_destructive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`
360
+
- **For additive changes**: the [model's `on_additive_change` value](../guides/incremental_time.md#schema-changes) (if present), the `on_additive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `allow`
361
+
362
+
If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's `--allow-destructive-model` selector to specify which models.
363
+
Similarly, if you want to temporarily allow additive changes to models configured with `on_additive_change=error`, use the `--allow-additive-model` selector.
364
+
365
+
For example, to allow additive changes to all models in the `analytics` schema:
366
+
```bash
367
+
sqlmesh plan --forward-only --allow-additive-model "analytics.*"
368
+
```
369
+
370
+
Or to allow additive changes to multiple specific models:
371
+
```bash
372
+
sqlmesh plan --forward-only --allow-additive-model "sales.revenue_model" --allow-additive-model "marketing.campaign_model"
373
+
```
374
+
375
+
Learn more about model selectors [here](../guides/model_selection.md).
360
376
361
377
### Effective date
362
378
Changes that are part of the forward-only plan can also be applied retroactively to the production environment by specifying the effective date:
Alternatively, all the changes contained in a *specific plan* can be classified as forward-only with a flag: `sqlmesh plan --forward-only`. A subsequent plan that did not include the forward-only flag would fully refresh the model's physical table. Learn more about forward-only plans [here](../concepts/plans.md#forward-only-plans).
161
161
162
-
### Destructive changes
162
+
### Schema changes
163
163
164
-
Some model changes destroy existing data in a table. Dropping a column from the model is the most direct cause, but changing a column's data type (such as casting a column from a `STRING`to `INTEGER`) can also require a drop. (Whether or not a specific change requires dropping a column may differ across SQL engines.)
164
+
When SQLMesh processes forward-only changes to incremental models, it compares the model's new schema with the existing physical table schema to detect potential data loss or compatibility issues. SQLMesh categorizes schema changes into two types:
165
165
166
-
Forward-only models are used to retain existing data. Before executing forward-only changes to incremental models, SQLMesh performs a check to determine if existing data will be destroyed.
166
+
#### Destructive changes
167
+
Some model changes destroy existing data in a table. Examples include:
167
168
168
-
The check is performed at plan time based on the model definition. SQLMesh may not be able to resolve all of a model's column data types and complete the check, so the check is performed again at run time based on the physical tables underlying the model.
169
+
-**Dropping a column** from the model
170
+
-**Renaming a column** without proper aliasing
171
+
-**Modifying column constraints** in ways that could cause data loss
172
+
173
+
Whether a specific change is destructive may differ across SQL engines based on their schema evolution capabilities.
174
+
175
+
#### Additive changes
176
+
Additive changes are any changes that aren't categorized as destructive. A simple examples would be adding a column to a table.
177
+
178
+
SQLMesh performs schema change detection at plan time based on the model definition. If SQLMesh cannot resolve all of a model's column data types at plan time, the check is performed again at run time based on the physical tables underlying the model.
169
179
170
180
#### Changes to forward-only models
171
181
172
-
A model's `on_destructive_change`[configuration setting](../reference/model_configuration.md#incremental-models) determines what happens when SQLMesh detects a destructive change.
182
+
SQLMesh provides two configuration settings to control how schema changes are handled:
183
+
184
+
-**`on_destructive_change`** - Controls behavior for destructive schema changes
185
+
-**`on_additive_change`** - Controls behavior for additive schema changes
186
+
187
+
##### Configuration options
188
+
189
+
Both properties support four values:
190
+
191
+
-**`error`** (default for `on_destructive_change`): Stop execution and raise an error
192
+
-**`warn`**: Log a warning but proceed with the change
193
+
-**`allow`** (default for `on_additive_change`): Silently proceed with the change
194
+
-**`ignore`**: Skip the schema change check entirely for this change type
195
+
196
+
!!! warning "Ignore is Dangerous"
173
197
174
-
By default, SQLMesh will error so no data is lost. You can set `on_destructive_change` to `warn` or `allow` in the model's `MODEL` block to allow destructive changes.
198
+
`ignore` is dangerous since it can result in error or data loss. It likely should never be used but could be useful as an "escape-hatch" or a way to workaround unexpected behavior.
199
+
200
+
##### Destructive change handling
201
+
202
+
The `on_destructive_change`[configuration setting](../reference/model_configuration.md#incremental-models) determines what happens when SQLMesh detects a destructive change. By default, SQLMesh will error so no data is lost.
175
203
176
204
This example configures a model to silently `allow` destructive changes:
177
205
@@ -186,12 +214,97 @@ MODEL (
186
214
);
187
215
```
188
216
189
-
A default `on_destructive_change` value can be set for all incremental models that do not specify it themselves in the [model defaults configuration](../reference/model_configuration.md#model-defaults).
217
+
##### Additive change handling
218
+
219
+
The `on_additive_change` configuration setting determines what happens when SQLMesh detects an additive change like adding new columns. By default, SQLMesh allows these changes since they don't destroy existing data.
220
+
221
+
This example configures a model to raise an error for additive changes (useful for strict schema control):
222
+
223
+
```sql linenums="1"
224
+
MODEL (
225
+
name sqlmesh_example.new_model,
226
+
kind INCREMENTAL_BY_TIME_RANGE (
227
+
time_column model_time_column,
228
+
forward_only true,
229
+
on_additive_change error
230
+
),
231
+
);
232
+
```
233
+
234
+
##### Combining both settings
235
+
236
+
You can configure both settings together to have fine-grained control over schema evolution:
237
+
238
+
```sql linenums="1"
239
+
MODEL (
240
+
name sqlmesh_example.new_model,
241
+
kind INCREMENTAL_BY_TIME_RANGE (
242
+
time_column model_time_column,
243
+
forward_only true,
244
+
on_destructive_change warn, -- Warn but allow destructive changes
245
+
on_additive_change allow -- Silently allow new columns
246
+
),
247
+
);
248
+
```
249
+
250
+
!!! warning "Unusual combinations"
251
+
252
+
SQLMesh will warn about unusual combinations such as allowing destructive changes while erroring on additive changes, since this indicates a potentially inconsistent schema change policy.
253
+
254
+
##### Model defaults
255
+
256
+
Default values for both `on_destructive_change` and `on_additive_change` can be set for all incremental models in the [model defaults configuration](../reference/model_configuration.md#model-defaults).
257
+
258
+
##### Common use cases
259
+
260
+
Here are some common patterns for configuring schema change handling:
261
+
262
+
**Strict schema control** - Prevent any schema changes:
**Production safety** - Allow safe changes, warn about risky ones:
289
+
```sql linenums="1"
290
+
MODEL (
291
+
name sqlmesh_example.production_model,
292
+
kind INCREMENTAL_BY_TIME_RANGE (
293
+
time_column event_date,
294
+
forward_only true,
295
+
on_destructive_change warn, -- Warn about destructive changes
296
+
on_additive_change allow -- Silently allow new columns
297
+
),
298
+
);
299
+
```
190
300
191
301
#### Changes in forward-only plans
192
302
193
-
The SQLMesh `plan`[`--forward-only` option](../concepts/plans.md#forward-only-plans) treats all the plan's model changes as forward-only. When this option is specified, SQLMesh will check all modified incremental models for destructive schema changes, not just models configured with `forward_only true`.
303
+
The SQLMesh `plan`[`--forward-only` option](../concepts/plans.md#forward-only-plans) treats all the plan's model changes as forward-only. When this option is specified, SQLMesh will check all modified incremental models for both destructive and additive schema changes, not just models configured with `forward_only true`.
304
+
305
+
SQLMesh determines what to do for each model based on this setting hierarchy:
194
306
195
-
SQLMesh determines what to do for each model based on this setting hierarchy: the model's `on_destructive_change` value (if present), the `on_destructive_change`[model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`.
307
+
-**For destructive changes**: the model's `on_destructive_change` value (if present), the `on_destructive_change`[model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`
308
+
-**For additive changes**: the model's `on_additive_change` value (if present), the `on_additive_change`[model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `allow`
196
309
197
-
If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's [`--allow-destructive-model` selector](../concepts/plans.md#destructive-changes) to specify which models. Learn more about model selectors [here](../guides/model_selection.md).
310
+
If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's [`--allow-destructive-model` selector](../concepts/plans.md#destructive-changes) to specify which models. Similarly, if you want to temporarily allow additive changes to models configured with `on_additive_change=error`, use the [`--allow-additive-model` selector](../concepts/plans.md#destructive-changes). Learn more about model selectors [here](../guides/model_selection.md).
Copy file name to clipboardExpand all lines: docs/guides/model_selection.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
This guide describes how to select specific models to include in a SQLMesh plan, which can be useful when modifying a subset of the models in a SQLMesh project.
4
4
5
-
Note: the selector syntax described below is also used for the SQLMesh `plan`[`--allow-destructive-model`selector](../concepts/plans.md#destructive-changes) and for the `table_diff` command to [diff a selection of models](./tablediff.md#diffing-multiple-models-across-environments).
5
+
Note: the selector syntax described below is also used for the SQLMesh `plan`[`--allow-destructive-model`and `--allow-additive-model` selectors](../concepts/plans.md#destructive-changes) and for the `table_diff` command to [diff a selection of models](./tablediff.md#diffing-multiple-models-across-environments).
Copy file name to clipboardExpand all lines: docs/integrations/dbt.md
+9-9Lines changed: 9 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -211,18 +211,18 @@ Similarly, the [allow_partials](../concepts/models/overview.md#allow_partials) p
211
211
212
212
#### on_schema_change
213
213
214
-
SQLMesh automatically detects destructive schema changes to [forward-only incremental models](../guides/incremental_time.md#forward-only-models) and to all incremental models in [forward-only plans](../concepts/plans.md#destructive-changes).
214
+
SQLMesh automatically detects both destructive and additive schema changes to [forward-only incremental models](../guides/incremental_time.md#forward-only-models) and to all incremental models in [forward-only plans](../concepts/plans.md#destructive-changes).
215
215
216
-
A model's [`on_destructive_change`setting](../guides/incremental_time.md#destructive-changes) determines whether it errors (default), warns, or silently allowsthe changes. SQLMesh always allows non-destructive forward-only schema changes, such as adding or casting a column in place.
216
+
A model's [`on_destructive_change`and `on_additive_change` settings](../guides/incremental_time.md#schema-changes) determine whether it errors, warns, silently allows, or ignores the changes. SQLMesh provides fine-grained control over both destructive changes (like dropping columns) and additive changes (like adding new columns).
217
217
218
-
`on_schema_change` configuration values are mapped to these SQLMesh `on_destructive_change` values:
218
+
`on_schema_change` configuration values are mapped to these SQLMesh settings:
0 commit comments