Skip to content

Commit bd67ec2

Browse files
committed
feat!: add support for on_additive_change
1 parent 49b5574 commit bd67ec2

28 files changed

+2005
-108
lines changed

docs/concepts/models/overview.md

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -507,11 +507,18 @@ Some properties are only available in specific model kinds - see the [model conf
507507
: Set this to true to indicate that all changes to this model should be [forward-only](../plans.md#forward-only-plans).
508508

509509
### on_destructive_change
510-
: What should happen when a change to a [forward-only model](../../guides/incremental_time.md#forward-only-models) or incremental model in a [forward-only plan](../plans.md#forward-only-plans) causes a destructive modification to the table schema (i.e., requires dropping an existing column).
510+
: What should happen when a change to a [forward-only model](../../guides/incremental_time.md#forward-only-models) or incremental model in a [forward-only plan](../plans.md#forward-only-plans) causes a destructive modification to the table schema (i.e., requires dropping an existing column or modifying column constraints in ways that could cause data loss).
511511

512512
SQLMesh checks for destructive changes at plan time based on the model definition and run time based on the model's underlying physical tables.
513513

514-
Must be one of the following values: `allow`, `warn`, or `error` (default).
514+
Must be one of the following values: `allow`, `warn`, `error` (default), or `ignore`.
515+
516+
### on_additive_change
517+
: What should happen when a change to a [forward-only model](../../guides/incremental_time.md#forward-only-models) or incremental model in a [forward-only plan](../plans.md#forward-only-plans) causes an additive modification to the table schema (i.e., adding new columns, modifying column data types in compatible ways, ect.).
518+
519+
SQLMesh checks for additive changes at plan time based on the model definition and run time based on the model's underlying physical tables.
520+
521+
Must be one of the following values: `allow` (default), `warn`, `error`, or `ignore`.
515522

516523
### disable_restatement
517524
: Set this to true to indicate that [data restatement](../plans.md#restatement-plans) is disabled for this model.

docs/concepts/plans.md

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -354,9 +354,25 @@ Some model changes destroy existing data in a table. SQLMesh automatically detec
354354
355355
Forward-only plans treats all of the plan's model changes as forward-only. In these plans, SQLMesh will check all modified incremental models for destructive schema changes, not just forward-only models.
356356
357-
SQLMesh determines what to do for each model based on this setting hierarchy: the [model's `on_destructive_change` value](../guides/incremental_time.md#destructive-changes) (if present), the `on_destructive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`.
357+
SQLMesh determines what to do for each model based on this setting hierarchy:
358358
359-
If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's `--allow-destructive-model` selector to specify which models. Learn more about model selectors [here](../guides/model_selection.md).
359+
- **For destructive changes**: the [model's `on_destructive_change` value](../guides/incremental_time.md#schema-changes) (if present), the `on_destructive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`
360+
- **For additive changes**: the [model's `on_additive_change` value](../guides/incremental_time.md#schema-changes) (if present), the `on_additive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `allow`
361+
362+
If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's `--allow-destructive-model` selector to specify which models.
363+
Similarly, if you want to temporarily allow additive changes to models configured with `on_additive_change=error`, use the `--allow-additive-model` selector.
364+
365+
For example, to allow additive changes to all models in the `analytics` schema:
366+
```bash
367+
sqlmesh plan --forward-only --allow-additive-model "analytics.*"
368+
```
369+
370+
Or to allow additive changes to multiple specific models:
371+
```bash
372+
sqlmesh plan --forward-only --allow-additive-model "sales.revenue_model" --allow-additive-model "marketing.campaign_model"
373+
```
374+
375+
Learn more about model selectors [here](../guides/model_selection.md).
360376
361377
### Effective date
362378
Changes that are part of the forward-only plan can also be applied retroactively to the production environment by specifying the effective date:

docs/guides/incremental_time.md

Lines changed: 123 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -159,19 +159,47 @@ WHERE
159159

160160
Alternatively, all the changes contained in a *specific plan* can be classified as forward-only with a flag: `sqlmesh plan --forward-only`. A subsequent plan that did not include the forward-only flag would fully refresh the model's physical table. Learn more about forward-only plans [here](../concepts/plans.md#forward-only-plans).
161161

162-
### Destructive changes
162+
### Schema changes
163163

164-
Some model changes destroy existing data in a table. Dropping a column from the model is the most direct cause, but changing a column's data type (such as casting a column from a `STRING` to `INTEGER`) can also require a drop. (Whether or not a specific change requires dropping a column may differ across SQL engines.)
164+
When SQLMesh processes forward-only changes to incremental models, it compares the model's new schema with the existing physical table schema to detect potential data loss or compatibility issues. SQLMesh categorizes schema changes into two types:
165165

166-
Forward-only models are used to retain existing data. Before executing forward-only changes to incremental models, SQLMesh performs a check to determine if existing data will be destroyed.
166+
#### Destructive changes
167+
Some model changes destroy existing data in a table. Examples include:
167168

168-
The check is performed at plan time based on the model definition. SQLMesh may not be able to resolve all of a model's column data types and complete the check, so the check is performed again at run time based on the physical tables underlying the model.
169+
- **Dropping a column** from the model
170+
- **Renaming a column** without proper aliasing
171+
- **Modifying column constraints** in ways that could cause data loss
172+
173+
Whether a specific change is destructive may differ across SQL engines based on their schema evolution capabilities.
174+
175+
#### Additive changes
176+
Additive changes are any changes that aren't categorized as destructive. A simple examples would be adding a column to a table.
177+
178+
SQLMesh performs schema change detection at plan time based on the model definition. If SQLMesh cannot resolve all of a model's column data types at plan time, the check is performed again at run time based on the physical tables underlying the model.
169179

170180
#### Changes to forward-only models
171181

172-
A model's `on_destructive_change` [configuration setting](../reference/model_configuration.md#incremental-models) determines what happens when SQLMesh detects a destructive change.
182+
SQLMesh provides two configuration settings to control how schema changes are handled:
183+
184+
- **`on_destructive_change`** - Controls behavior for destructive schema changes
185+
- **`on_additive_change`** - Controls behavior for additive schema changes
186+
187+
##### Configuration options
188+
189+
Both properties support four values:
190+
191+
- **`error`** (default for `on_destructive_change`): Stop execution and raise an error
192+
- **`warn`**: Log a warning but proceed with the change
193+
- **`allow`** (default for `on_additive_change`): Silently proceed with the change
194+
- **`ignore`**: Skip the schema change check entirely for this change type
195+
196+
!!! warning "Ignore is Dangerous"
173197

174-
By default, SQLMesh will error so no data is lost. You can set `on_destructive_change` to `warn` or `allow` in the model's `MODEL` block to allow destructive changes.
198+
`ignore` is dangerous since it can result in error or data loss. It likely should never be used but could be useful as an "escape-hatch" or a way to workaround unexpected behavior.
199+
200+
##### Destructive change handling
201+
202+
The `on_destructive_change` [configuration setting](../reference/model_configuration.md#incremental-models) determines what happens when SQLMesh detects a destructive change. By default, SQLMesh will error so no data is lost.
175203

176204
This example configures a model to silently `allow` destructive changes:
177205

@@ -186,12 +214,97 @@ MODEL (
186214
);
187215
```
188216

189-
A default `on_destructive_change` value can be set for all incremental models that do not specify it themselves in the [model defaults configuration](../reference/model_configuration.md#model-defaults).
217+
##### Additive change handling
218+
219+
The `on_additive_change` configuration setting determines what happens when SQLMesh detects an additive change like adding new columns. By default, SQLMesh allows these changes since they don't destroy existing data.
220+
221+
This example configures a model to raise an error for additive changes (useful for strict schema control):
222+
223+
``` sql linenums="1"
224+
MODEL (
225+
name sqlmesh_example.new_model,
226+
kind INCREMENTAL_BY_TIME_RANGE (
227+
time_column model_time_column,
228+
forward_only true,
229+
on_additive_change error
230+
),
231+
);
232+
```
233+
234+
##### Combining both settings
235+
236+
You can configure both settings together to have fine-grained control over schema evolution:
237+
238+
``` sql linenums="1"
239+
MODEL (
240+
name sqlmesh_example.new_model,
241+
kind INCREMENTAL_BY_TIME_RANGE (
242+
time_column model_time_column,
243+
forward_only true,
244+
on_destructive_change warn, -- Warn but allow destructive changes
245+
on_additive_change allow -- Silently allow new columns
246+
),
247+
);
248+
```
249+
250+
!!! warning "Unusual combinations"
251+
252+
SQLMesh will warn about unusual combinations such as allowing destructive changes while erroring on additive changes, since this indicates a potentially inconsistent schema change policy.
253+
254+
##### Model defaults
255+
256+
Default values for both `on_destructive_change` and `on_additive_change` can be set for all incremental models in the [model defaults configuration](../reference/model_configuration.md#model-defaults).
257+
258+
##### Common use cases
259+
260+
Here are some common patterns for configuring schema change handling:
261+
262+
**Strict schema control** - Prevent any schema changes:
263+
```sql linenums="1"
264+
MODEL (
265+
name sqlmesh_example.strict_model,
266+
kind INCREMENTAL_BY_TIME_RANGE (
267+
time_column event_date,
268+
forward_only true,
269+
on_destructive_change error, -- Block destructive changes
270+
on_additive_change error -- Block even new columns
271+
),
272+
);
273+
```
274+
275+
**Permissive development model** - Allow all schema changes:
276+
```sql linenums="1"
277+
MODEL (
278+
name sqlmesh_example.dev_model,
279+
kind INCREMENTAL_BY_TIME_RANGE (
280+
time_column event_date,
281+
forward_only true,
282+
on_destructive_change allow, -- Allow dropping columns
283+
on_additive_change allow -- Allow new columns
284+
),
285+
);
286+
```
287+
288+
**Production safety** - Allow safe changes, warn about risky ones:
289+
```sql linenums="1"
290+
MODEL (
291+
name sqlmesh_example.production_model,
292+
kind INCREMENTAL_BY_TIME_RANGE (
293+
time_column event_date,
294+
forward_only true,
295+
on_destructive_change warn, -- Warn about destructive changes
296+
on_additive_change allow -- Silently allow new columns
297+
),
298+
);
299+
```
190300

191301
#### Changes in forward-only plans
192302

193-
The SQLMesh `plan` [`--forward-only` option](../concepts/plans.md#forward-only-plans) treats all the plan's model changes as forward-only. When this option is specified, SQLMesh will check all modified incremental models for destructive schema changes, not just models configured with `forward_only true`.
303+
The SQLMesh `plan` [`--forward-only` option](../concepts/plans.md#forward-only-plans) treats all the plan's model changes as forward-only. When this option is specified, SQLMesh will check all modified incremental models for both destructive and additive schema changes, not just models configured with `forward_only true`.
304+
305+
SQLMesh determines what to do for each model based on this setting hierarchy:
194306

195-
SQLMesh determines what to do for each model based on this setting hierarchy: the model's `on_destructive_change` value (if present), the `on_destructive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`.
307+
- **For destructive changes**: the model's `on_destructive_change` value (if present), the `on_destructive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `error`
308+
- **For additive changes**: the model's `on_additive_change` value (if present), the `on_additive_change` [model defaults](../reference/model_configuration.md#model-defaults) value (if present), and the SQLMesh global default of `allow`
196309

197-
If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's [`--allow-destructive-model` selector](../concepts/plans.md#destructive-changes) to specify which models. Learn more about model selectors [here](../guides/model_selection.md).
310+
If you want to temporarily allow destructive changes to models that don't allow them, use the `plan` command's [`--allow-destructive-model` selector](../concepts/plans.md#destructive-changes) to specify which models. Similarly, if you want to temporarily allow additive changes to models configured with `on_additive_change=error`, use the [`--allow-additive-model` selector](../concepts/plans.md#destructive-changes). Learn more about model selectors [here](../guides/model_selection.md).

docs/guides/model_selection.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
This guide describes how to select specific models to include in a SQLMesh plan, which can be useful when modifying a subset of the models in a SQLMesh project.
44

5-
Note: the selector syntax described below is also used for the SQLMesh `plan` [`--allow-destructive-model` selector](../concepts/plans.md#destructive-changes) and for the `table_diff` command to [diff a selection of models](./tablediff.md#diffing-multiple-models-across-environments).
5+
Note: the selector syntax described below is also used for the SQLMesh `plan` [`--allow-destructive-model` and `--allow-additive-model` selectors](../concepts/plans.md#destructive-changes) and for the `table_diff` command to [diff a selection of models](./tablediff.md#diffing-multiple-models-across-environments).
66

77
## Background
88

docs/integrations/dbt.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -211,18 +211,18 @@ Similarly, the [allow_partials](../concepts/models/overview.md#allow_partials) p
211211
212212
#### on_schema_change
213213
214-
SQLMesh automatically detects destructive schema changes to [forward-only incremental models](../guides/incremental_time.md#forward-only-models) and to all incremental models in [forward-only plans](../concepts/plans.md#destructive-changes).
214+
SQLMesh automatically detects both destructive and additive schema changes to [forward-only incremental models](../guides/incremental_time.md#forward-only-models) and to all incremental models in [forward-only plans](../concepts/plans.md#destructive-changes).
215215
216-
A model's [`on_destructive_change` setting](../guides/incremental_time.md#destructive-changes) determines whether it errors (default), warns, or silently allows the changes. SQLMesh always allows non-destructive forward-only schema changes, such as adding or casting a column in place.
216+
A model's [`on_destructive_change` and `on_additive_change` settings](../guides/incremental_time.md#schema-changes) determine whether it errors, warns, silently allows, or ignores the changes. SQLMesh provides fine-grained control over both destructive changes (like dropping columns) and additive changes (like adding new columns).
217217
218-
`on_schema_change` configuration values are mapped to these SQLMesh `on_destructive_change` values:
218+
`on_schema_change` configuration values are mapped to these SQLMesh settings:
219219
220-
| `on_schema_change` | SQLMesh `on_destructive_change` |
221-
| ------------------ | ------------------------------- |
222-
| ignore | warn |
223-
| append_new_columns | warn |
224-
| sync_all_columns | allow |
225-
| fail | error |
220+
| `on_schema_change` | SQLMesh `on_destructive_change` | SQLMesh `on_additive_change` |
221+
|--------------------|---------------------------------|------------------------------|
222+
| ignore | ignore | ignore |
223+
| fail | error | error |
224+
| append_new_columns | ignore | allow |
225+
| sync_all_columns | allow | allow |
226226
227227
228228
## Snapshot support

docs/reference/cli.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -367,6 +367,8 @@ Options:
367367
--forward-only Create a plan for forward-only changes.
368368
--allow-destructive-model TEXT Allow destructive forward-only changes to
369369
models whose names match the expression.
370+
--allow-additive-model TEXT Allow additive forward-only changes to
371+
models whose names match the expression.
370372
--effective-from TEXT The effective date from which to apply
371373
forward-only changes on production.
372374
--no-prompts Disable interactive prompts for the backfill

0 commit comments

Comments
 (0)