Merge remote-tracking branch 'upstream/main' into feat/add-fabric-engine

Author: mattiasthalen

.pre-commit-config.yaml

@@ -23,3 +23,8 @@ repos:
         files: *files
         require_serial: true
         exclude: ^(tests/fixtures)
+      - id: valid migrations
+        name: valid migrations
+        entry: tooling/validating_migration_numbers.sh
+        language: system
+        pass_filenames: false

docs/concepts/models/python_models.md

@@ -102,6 +102,8 @@ For example, pre/post-statements might modify settings or create indexes. Howeve
 
 You can set the `pre_statements` and `post_statements` arguments to a list of SQL strings, SQLGlot expressions, or macro calls to define the model's pre/post-statements.
 
+**Project-level defaults:** You can also define pre/post-statements at the project level using `model_defaults` in your configuration. These will be applied to all models in your project and merged with any model-specific statements. Default statements are executed first, followed by model-specific statements. Learn more about this in the [model configuration reference](../../reference/model_configuration.md#model-defaults).
+
 ``` python linenums="1" hl_lines="8-12"
 @model(
     "db.test_model",
@@ -182,6 +184,8 @@ These can be used, for example, to grant privileges on views of the virtual laye
 
 Similar to pre/post-statements you can set the `on_virtual_update` argument in the `@model` decorator to a list of SQL strings, SQLGlot expressions, or macro calls.
 
+**Project-level defaults:** You can also define on-virtual-update statements at the project level using `model_defaults` in your configuration. These will be applied to all models in your project (including Python models) and merged with any model-specific statements. Default statements are executed first, followed by model-specific statements. Learn more about this in the [model configuration reference](../../reference/model_configuration.md#model-defaults).
+
 ``` python linenums="1" hl_lines="8"
 @model(
     "db.test_model",

docs/concepts/models/seed_models.md

@@ -203,6 +203,8 @@ ALTER SESSION SET TIMEZONE = 'PST';
 
 Seed models also support on-virtual-update statements, which are executed after the completion of the [Virtual Update](#virtual-update).
 
+**Project-level defaults:** You can also define on-virtual-update statements at the project level using `model_defaults` in your configuration. These will be applied to all models in your project (including seed models) and merged with any model-specific statements. Default statements are executed first, followed by model-specific statements. Learn more about this in the [model configuration reference](../../reference/model_configuration.md#model-defaults).
+
 These must be enclosed within an `ON_VIRTUAL_UPDATE_BEGIN;` ...; `ON_VIRTUAL_UPDATE_END;` block:
 
 ```sql linenums="1" hl_lines="8-13"

docs/concepts/models/sql_models.md

@@ -67,6 +67,8 @@ For example, pre/post-statements might modify settings or create a table index.
 
 Pre/post-statements are just standard SQL commands located before/after the model query. They must end with a semi-colon, and the model query must end with a semi-colon if a post-statement is present. The [example above](#example) contains both pre- and post-statements.
 
+**Project-level defaults:** You can also define pre/post-statements at the project level using `model_defaults` in your configuration. These will be applied to all models in your project and merged with any model-specific statements. Default statements are executed first, followed by model-specific statements. Learn more about this in the [model configuration reference](../../reference/model_configuration.md#model-defaults).
+
 !!! warning
 
     Pre/post-statements are evaluated twice: when a model's table is created and when its query logic is evaluated. Executing statements more than once can have unintended side-effects, so you can [conditionally execute](../macros/sqlmesh_macros.md#prepost-statements) them based on SQLMesh's [runtime stage](../macros/macro_variables.md#runtime-variables).
@@ -97,6 +99,8 @@ The optional on-virtual-update statements allow you to execute SQL commands afte
 
 These can be used, for example, to grant privileges on views of the virtual layer.
 
+**Project-level defaults:** You can also define on-virtual-update statements at the project level using `model_defaults` in your configuration. These will be applied to all models in your project and merged with any model-specific statements. Default statements are executed first, followed by model-specific statements. Learn more about this in the [model configuration reference](../../reference/model_configuration.md#model-defaults).
+
 These SQL statements must be enclosed within an `ON_VIRTUAL_UPDATE_BEGIN;` ...; `ON_VIRTUAL_UPDATE_END;` block like this:
 
 ```sql linenums="1" hl_lines="10-15"

docs/guides/configuration.md

@@ -320,10 +320,14 @@ The cache directory is automatically created if it doesn't exist. You can clear
 
 SQLMesh creates schemas, physical tables, and views in the data warehouse/engine. Learn more about why and how SQLMesh creates schema in the ["Why does SQLMesh create schemas?" FAQ](../faq/faq.md#schema-question).
 
-The default SQLMesh behavior described in the FAQ is appropriate for most deployments, but you can override where SQLMesh creates physical tables and views with the `physical_schema_mapping`, `environment_suffix_target`, and `environment_catalog_mapping` configuration options. These options are in the [environments](../reference/configuration.md#environments) section of the configuration reference page.
+The default SQLMesh behavior described in the FAQ is appropriate for most deployments, but you can override *where* SQLMesh creates physical tables and views with the `physical_schema_mapping`, `environment_suffix_target`, and `environment_catalog_mapping` configuration options.
+
+You can also override *what* the physical tables are called by using the `physical_table_naming_convention` option. 
+
+These options are in the [environments](../reference/configuration.md#environments) section of the configuration reference page.
 
 #### Physical table schemas
-By default, SQLMesh creates physical tables for a model with a naming convention of `sqlmesh__[model schema]`.
+By default, SQLMesh creates physical schemas for a model with a naming convention of `sqlmesh__[model schema]`.
 
 This can be overridden on a per-schema basis using the `physical_schema_mapping` option, which removes the `sqlmesh__` prefix and uses the [regex pattern](https://docs.python.org/3/library/re.html#regular-expression-syntax) you provide to map the schemas defined in your model to their corresponding physical schemas.
 
@@ -436,6 +440,104 @@ Given the example of a model called `my_schema.users` with a default catalog of
     - Using `environment_suffix_target: catalog` only works on engines that support querying across different catalogs. If your engine does not support cross-catalog queries then you will need to use `environment_suffix_target: schema` or `environment_suffix_target: table` instead.
     - Automatic catalog creation is not supported on all engines even if they support cross-catalog queries. For engines where it is not supported, the catalogs must be managed externally from SQLMesh and exist prior to invoking SQLMesh.
 
+#### Physical table naming convention
+
+Out of the box, SQLMesh has the following defaults set:
+
+ - `environment_suffix_target: schema`
+ - `physical_table_naming_convention: schema_and_table`
+ - no `physical_schema_mapping` overrides, so a `sqlmesh__<model schema>` physical schema will be created for each model schema
+
+This means that given a catalog of `warehouse` and a model named `finance_mart.transaction_events_over_threshold`, SQLMesh will create physical tables using the following convention:
+
+```
+# <catalog>.sqlmesh__<schema>.<schema>__<table>__<fingerprint>
+
+warehouse.sqlmesh__finance_mart.finance_mart__transaction_events_over_threshold__<fingerprint>
+```
+
+This deliberately contains some redundancy with the *model* schema as it's repeated at the physical layer in both the physical schema name as well as the physical table name.
+
+This default exists to make the physical table names portable between different configurations. If you were to define a `physical_schema_mapping` that maps all models to the same physical schema, since the model schema is included in the table name as well, there are no naming conflicts.
+
+##### Table only
+
+Some engines have object name length limitations which cause them to [silently truncate](https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS) table and view names that exceed this limit. This behaviour breaks SQLMesh, so we raise a runtime error if we detect the engine would silently truncate the name of the table we are trying to create.
+
+Having redundancy in the physical table names does reduce the number of characters that can be utilised in model names. To increase the number of characters available to model names, you can use `physical_table_naming_convention` like so:
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    physical_table_naming_convention: table_only
+    ```
+
+=== "Python"
+
+    ```python linenums="1"
+    from sqlmesh.core.config import Config, ModelDefaultsConfig, TableNamingConvention
+
+    config = Config(
+        model_defaults=ModelDefaultsConfig(dialect=<dialect>),
+        physical_table_naming_convention=TableNamingConvention.TABLE_ONLY,
+    )
+    ```
+
+This will cause SQLMesh to omit the model schema from the table name and generate physical names that look like (using the above example):
+```
+# <catalog>.sqlmesh__<schema>.<table>__<fingerprint>
+
+warehouse.sqlmesh__finance_mart.transaction_events_over_threshold__<fingerprint>
+```
+
+Notice that the model schema name is no longer part of the physical table name. This allows for slightly longer model names on engines with low identifier length limits, which may be useful for your project.
+
+In this configuration, it is your responsibility to ensure that any schema overrides in `physical_schema_mapping` result in each model schema getting mapped to a unique physical schema.
+
+For example, the following configuration will cause **data corruption**:
+
+```yaml
+physical_table_naming_convention: table_only
+physical_schema_mapping:
+  '.*': sqlmesh
+```
+
+This is because every model schema is mapped to the same physical schema but the model schema name is omitted from the physical table name.
+
+##### MD5 hash
+
+If you *still* need more characters, you can set `physical_table_naming_convention: hash_md5` like so:
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    physical_table_naming_convention: hash_md5
+    ```
+
+=== "Python"
+
+    ```python linenums="1"
+    from sqlmesh.core.config import Config, ModelDefaultsConfig, TableNamingConvention
+
+    config = Config(
+        model_defaults=ModelDefaultsConfig(dialect=<dialect>),
+        physical_table_naming_convention=TableNamingConvention.HASH_MD5,
+    )
+    ```
+
+This will cause SQLMesh generate physical names that are always 45-50 characters in length and look something like:
+
+```
+# sqlmesh_md5__<hash of what we would have generated using 'schema_and_table'>
+
+sqlmesh_md5__d3b07384d113edec49eaa6238ad5ff00
+
+# or, for a dev preview
+sqlmesh_md5__d3b07384d113edec49eaa6238ad5ff00__dev
+```
+
+This has a downside that now it's much more difficult to determine which table corresponds to which model by just looking at the database with a SQL client. However, the table names have a predictable length so there are no longer any surprises with identfiers exceeding the max length at the physical layer.
+
 #### Environment view catalogs
 
 By default, SQLMesh creates an environment view in the same [catalog](../concepts/glossary.md#catalog) as the physical table the view points to. The physical table's catalog is determined by either the catalog specified in the model name or the default catalog defined in the connection.