Feat: Allow some control of table naming at the physical layer

Author: erindru

docs/guides/configuration.md

@@ -320,10 +320,14 @@ The cache directory is automatically created if it doesn't exist. You can clear
 
 SQLMesh creates schemas, physical tables, and views in the data warehouse/engine. Learn more about why and how SQLMesh creates schema in the ["Why does SQLMesh create schemas?" FAQ](../faq/faq.md#schema-question).
 
-The default SQLMesh behavior described in the FAQ is appropriate for most deployments, but you can override where SQLMesh creates physical tables and views with the `physical_schema_mapping`, `environment_suffix_target`, and `environment_catalog_mapping` configuration options. These options are in the [environments](../reference/configuration.md#environments) section of the configuration reference page.
+The default SQLMesh behavior described in the FAQ is appropriate for most deployments, but you can override *where* SQLMesh creates physical tables and views with the `physical_schema_mapping`, `environment_suffix_target`, and `environment_catalog_mapping` configuration options.
+
+You can also override *what* the physical tables are called by using the `physical_table_naming_convention` option. 
+
+These options are in the [environments](../reference/configuration.md#environments) section of the configuration reference page.
 
 #### Physical table schemas
-By default, SQLMesh creates physical tables for a model with a naming convention of `sqlmesh__[model schema]`.
+By default, SQLMesh creates physical schemas for a model with a naming convention of `sqlmesh__[model schema]`.
 
 This can be overridden on a per-schema basis using the `physical_schema_mapping` option, which removes the `sqlmesh__` prefix and uses the [regex pattern](https://docs.python.org/3/library/re.html#regular-expression-syntax) you provide to map the schemas defined in your model to their corresponding physical schemas.
 
@@ -436,6 +440,89 @@ Given the example of a model called `my_schema.users` with a default catalog of
     - Using `environment_suffix_target: catalog` only works on engines that support querying across different catalogs. If your engine does not support cross-catalog queries then you will need to use `environment_suffix_target: schema` or `environment_suffix_target: table` instead.
     - Automatic catalog creation is not supported on all engines even if they support cross-catalog queries. For engines where it is not supported, the catalogs must be managed externally from SQLMesh and exist prior to invoking SQLMesh.
 
+#### Physical table naming convention
+
+Out of the box, SQLMesh has the following defaults set:
+
+ - `environment_suffix_target: schema`
+ - `physical_table_naming_convention: schema_and_table`
+
+Given a catalog of `warehouse` and a model named `finance_mart.transaction_events_over_threshold`, this causes SQLMesh to create physical tables using the following convention:
+
+```
+# <catalog>.sqlmesh__<schema>.<schema>__<table>__<fingerprint>
+
+warehouse.sqlmesh__finance_mart.finance_mart__transaction_events_over_threshold__<fingerprint>
+```
+
+This deliberately contains some redundancy with the *model* schema as it's repeated at the physical layer in both the physical schema name as well as the physical table name.
+
+##### Table only
+
+Some engines have object name length limitations which cause them to [silently truncate](https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS) table and view names that exceed this limit. This behaviour breaks SQLMesh, so we raise a runtime error if we detect the engine would silently truncate the name of the table we are trying to create.
+
+Having redundancy in the physical table names does reduce the number of characters that can be utilised in model names. To increase the number of characters available to model names, you can use `physical_table_naming_convention` like so:
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    physical_table_naming_convention: table_only
+    ```
+
+=== "Python"
+
+    ```python linenums="1"
+    from sqlmesh.core.config import Config, ModelDefaultsConfig, TableNamingConvention
+
+    config = Config(
+        model_defaults=ModelDefaultsConfig(dialect=<dialect>),
+        physical_table_naming_convention=TableNamingConvention.TABLE_ONLY,
+    )
+    ```
+
+This will cause SQLMesh to omit the model schema from the table name and generate physical names that look like (using the above example):
+```
+# <catalog>.sqlmesh__<schema>.<table>__<fingerprint>
+
+warehouse.sqlmesh__finance_mart.transaction_events_over_threshold__<fingerprint>
+```
+
+Notice that the model schema name is no longer part of the physical table name. This allows for slightly longer model names on engines with low identifier length limits, which may be useful for your project.
+
+##### MD5 hash
+
+If you *still* need more characters, you can set `physical_table_naming_convention: hash_md5` like so:
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    physical_table_naming_convention: hash_md5
+    ```
+
+=== "Python"
+
+    ```python linenums="1"
+    from sqlmesh.core.config import Config, ModelDefaultsConfig, TableNamingConvention
+
+    config = Config(
+        model_defaults=ModelDefaultsConfig(dialect=<dialect>),
+        physical_table_naming_convention=TableNamingConvention.HASH_MD5,
+    )
+    ```
+
+This will cause SQLMesh generate physical names that are always 45-50 characters in length and look something like:
+
+```
+# sqlmesh_md5__<hash of what we would have generated using 'schema_and_table'>
+
+sqlmesh_md5__d3b07384d113edec49eaa6238ad5ff00
+
+# or, for a dev preview
+sqlmesh_md5__d3b07384d113edec49eaa6238ad5ff00__dev
+```
+
+This has a downside that now it's much more difficult to determine which table corresponds to which model by just looking at the database with a SQL client. However, the table names now have a predictable length so there are no longer any surprises with identfiers exceeding the max length at the physical layer.
+
 #### Environment view catalogs
 
 By default, SQLMesh creates an environment view in the same [catalog](../concepts/glossary.md#catalog) as the physical table the view points to. The physical table's catalog is determined by either the catalog specified in the model name or the default catalog defined in the connection.

docs/reference/configuration.md

@@ -36,8 +36,9 @@ Configuration options for SQLMesh environment creation and promotion.
 | `physical_schema_override`    | (Deprecated) Use `physical_schema_mapping` instead. A mapping from model schema names to names of schemas in which physical tables for the corresponding models will be placed.                                                                                                                    | dict[string, string] | N        |
 | `physical_schema_mapping`     | A mapping from regular expressions to names of schemas in which physical tables for the corresponding models [will be placed](../guides/configuration.md#physical-table-schemas). (Default physical schema name: `sqlmesh__[model schema]`)                                                        | dict[string, string] | N        |
 | `environment_suffix_target`   | Whether SQLMesh views should append their environment name to the `schema` or `table` - [additional details](../guides/configuration.md#view-schema-override). (Default: `schema`)                                                                                                                 | string               | N        |
-| `gateway_managed_virtual_layer`   | Whether SQLMesh views of the virtual layer will be created by the default gateway or model specified gateways - [additional details](../guides/multi_engine.md#gateway-managed-virtual-layer). (Default: False)                                                                                                                 | boolean               | N        |
-| `infer_python_dependencies`   | Whether SQLMesh will statically analyze Python code to automatically infer Python package requirements. (Default: True)                                                                                                                 | boolean               | N        |
+| `physical_table_naming_convention`| Sets which parts of the model name are included in the physical table names. Options are `schema_and_table` or `table_only` - [additional details](../guides/configuration.md#physical-table-naming-convention). (Default: `schema_and_table`)                                                 | string               | N        |
+| `gateway_managed_virtual_layer`   | Whether SQLMesh views of the virtual layer will be created by the default gateway or model specified gateways - [additional details](../guides/multi_engine.md#gateway-managed-virtual-layer). (Default: False)                                                                                | boolean              | N        |
+| `infer_python_dependencies`   | Whether SQLMesh will statically analyze Python code to automatically infer Python package requirements. (Default: True)                                                                                                                                                                            | boolean              | N        |
 | `environment_catalog_mapping` | A mapping from regular expressions to catalog names. The catalog name is used to determine the target catalog for a given environment.                                                                                                                                                             | dict[string, string] | N        |
 | `log_limit`                   | The default number of logs to keep (Default: `20`)                                                                                                                                                                                                                                                 | int                  | N        |
 

sqlmesh/core/config/__init__.py

@@ -2,7 +2,10 @@
     AutoCategorizationMode as AutoCategorizationMode,
     CategorizerConfig as CategorizerConfig,
 )
-from sqlmesh.core.config.common import EnvironmentSuffixTarget as EnvironmentSuffixTarget
+from sqlmesh.core.config.common import (
+    EnvironmentSuffixTarget as EnvironmentSuffixTarget,
+    TableNamingConvention as TableNamingConvention,
+)
 from sqlmesh.core.config.connection import (
     AthenaConnectionConfig as AthenaConnectionConfig,
     BaseDuckDBConnectionConfig as BaseDuckDBConnectionConfig,

sqlmesh/core/config/common.py

@@ -49,6 +49,34 @@ def __repr__(self) -> str:
         return str(self)
 
 
+class TableNamingConvention(str, Enum):
+    # Causes table names at the physical layer to follow the convention:
+    # <schema-name>__<table-name>__<fingerprint>
+    SCHEMA_AND_TABLE = "schema_and_table"
+
+    # Causes table names at the physical layer to follow the convention:
+    # <table-name>__<fingerprint>
+    TABLE_ONLY = "table_only"
+
+    # Takes the table name that would be returned from SCHEMA_AND_TABLE and wraps it in md5()
+    # to generate a hash and prefixes the has with `sqlmesh_md5__`, for the following reasons:
+    # - at a glance, you can still see it's managed by sqlmesh and that md5 was used to generate the hash
+    # - unquoted identifiers that start with numbers can trip up DB engine parsers, so having a text prefix prevents this
+    # This causes table names at the physical layer to follow the convention:
+    # sqlmesh_md5__3b07384d113edec49eaa6238ad5ff00d
+    HASH_MD5 = "hash_md5"
+
+    @classproperty
+    def default(cls) -> TableNamingConvention:
+        return TableNamingConvention.SCHEMA_AND_TABLE
+
+    def __str__(self) -> str:
+        return self.name
+
+    def __repr__(self) -> str:
+        return str(self)
+
+
 def _concurrent_tasks_validator(v: t.Any) -> int:
     if isinstance(v, str):
         v = int(v)

sqlmesh/core/config/root.py

@@ -14,7 +14,7 @@
 from sqlmesh.cicd.config import CICDBotConfig
 from sqlmesh.core import constants as c
 from sqlmesh.core.console import get_console
-from sqlmesh.core.config import EnvironmentSuffixTarget
+from sqlmesh.core.config import EnvironmentSuffixTarget, TableNamingConvention
 from sqlmesh.core.config.base import BaseConfig, UpdateStrategy
 from sqlmesh.core.config.common import variables_validator, compile_regex_mapping
 from sqlmesh.core.config.connection import (
@@ -106,6 +106,7 @@ class Config(BaseConfig):
         model_defaults: Default values for model definitions.
         physical_schema_mapping: A mapping from regular expressions to names of schemas in which physical tables for corresponding models will be placed.
         environment_suffix_target: Indicates whether to append the environment name to the schema or table name.
+        physical_table_naming_convention: Indicates how tables should be named at the physical layer
         gateway_managed_virtual_layer: Whether the models' views in the virtual layer are created by the model-specific gateway rather than the default gateway.
         infer_python_dependencies: Whether to statically analyze Python code to automatically infer Python package requirements.
         environment_catalog_mapping: A mapping from regular expressions to catalog names. The catalog name is used to determine the target catalog for a given environment.
@@ -147,6 +148,7 @@ class Config(BaseConfig):
     environment_suffix_target: EnvironmentSuffixTarget = Field(
         default=EnvironmentSuffixTarget.default
     )
+    physical_table_naming_convention: t.Optional[TableNamingConvention] = None
     gateway_managed_virtual_layer: bool = False
     infer_python_dependencies: bool = True
     environment_catalog_mapping: RegexKeyDict = {}

sqlmesh/core/context.py

@@ -2904,9 +2904,11 @@ def _nodes_to_snapshots(self, nodes: t.Dict[str, Node]) -> t.Dict[str, Snapshot]
         fingerprint_cache: t.Dict[str, SnapshotFingerprint] = {}
 
         for node in nodes.values():
-            kwargs = {}
+            kwargs: t.Dict[str, t.Any] = {}
             if node.project in self._projects:
-                kwargs["ttl"] = self.config_for_node(node).snapshot_ttl
+                config = self.config_for_node(node)
+                kwargs["ttl"] = config.snapshot_ttl
+                kwargs["table_naming_convention"] = config.physical_table_naming_convention
 
             snapshot = Snapshot.from_node(
                 node,

sqlmesh/core/snapshot/definition.py

@@ -13,6 +13,7 @@
 from sqlglot import exp
 from sqlglot.optimizer.normalize_identifiers import normalize_identifiers
 
+from sqlmesh.core.config import TableNamingConvention
 from sqlmesh.core import constants as c
 from sqlmesh.core.audit import StandaloneAudit
 from sqlmesh.core.environment import EnvironmentSuffixTarget
@@ -44,7 +45,7 @@
     format_evaluated_code_exception,
     Executable,
 )
-from sqlmesh.utils.hashing import hash_data
+from sqlmesh.utils.hashing import hash_data, md5
 from sqlmesh.utils.pydantic import PydanticModel, field_validator
 
 if t.TYPE_CHECKING:
@@ -333,6 +334,7 @@ class SnapshotInfoMixin(ModelKindMixin):
     # This can be removed from this model once Pydantic 1 support is dropped (must remain in `Snapshot` though)
     base_table_name_override: t.Optional[str]
     dev_table_suffix: str
+    table_naming_convention: t.Optional[TableNamingConvention] = None
 
     @cached_property
     def identifier(self) -> str:
@@ -451,6 +453,7 @@ def _table_name(self, version: str, is_deployable: bool) -> str:
             version,
             catalog=self.fully_qualified_table.catalog,
             suffix=self.dev_table_suffix if is_dev_table else None,
+            naming_convention=self.table_naming_convention,
         )
 
     @property
@@ -580,6 +583,7 @@ class Snapshot(PydanticModel, SnapshotInfoMixin):
         migrated: Whether or not this snapshot has been created as a result of migration.
         unrestorable: Whether or not this snapshot can be used to revert its model to a previous version.
         next_auto_restatement_ts: The timestamp which indicates when is the next time this snapshot should be restated.
+        table_naming_convention: Convention to follow when generating the physical table name
     """
 
     name: str
@@ -605,6 +609,9 @@ class Snapshot(PydanticModel, SnapshotInfoMixin):
     base_table_name_override: t.Optional[str] = None
     next_auto_restatement_ts: t.Optional[int] = None
     dev_table_suffix: str = "dev"
+    table_naming_convention_: t.Optional[TableNamingConvention] = Field(
+        default=None, alias="table_naming_convention"
+    )
 
     @field_validator("ttl")
     @classmethod
@@ -656,6 +663,7 @@ def from_node(
         ttl: str = c.DEFAULT_SNAPSHOT_TTL,
         version: t.Optional[str] = None,
         cache: t.Optional[t.Dict[str, SnapshotFingerprint]] = None,
+        table_naming_convention: t.Optional[TableNamingConvention] = None,
     ) -> Snapshot:
         """Creates a new snapshot for a node.
 
@@ -666,6 +674,7 @@ def from_node(
             ttl: A TTL to determine how long orphaned (snapshots that are not promoted anywhere) should live.
             version: The version that a snapshot is associated with. Usually set during the planning phase.
             cache: Cache of node name to fingerprints.
+            table_naming_convention: Convention to follow when generating the physical table name
 
         Returns:
             The newly created snapshot.
@@ -697,6 +706,7 @@ def from_node(
             updated_ts=created_ts,
             ttl=ttl,
             version=version,
+            table_naming_convention=table_naming_convention,
         )
 
     def __eq__(self, other: t.Any) -> bool:
@@ -1206,6 +1216,7 @@ def table_info(self) -> SnapshotTableInfo:
             custom_materialization=custom_materialization,
             dev_table_suffix=self.dev_table_suffix,
             model_gateway=self.model_gateway,
+            table_naming_convention=self.table_naming_convention,  # type: ignore
         )
 
     @property
@@ -1568,14 +1579,41 @@ def table_name(
     version: str,
     catalog: t.Optional[str] = None,
     suffix: t.Optional[str] = None,
+    naming_convention: t.Optional[TableNamingConvention] = None,
 ) -> str:
     table = exp.to_table(name)
 
-    # bigquery projects usually have "-" in them which is illegal in the table name, so we aggressively prune
-    name = "__".join(sanitize_name(part.name) for part in table.parts)
+    naming_convention = naming_convention or TableNamingConvention.default
+
+    if naming_convention == TableNamingConvention.HASH_MD5:
+        # just take a MD5 hash of what we would have generated anyway using SCHEMA_AND_TABLE
+        value_to_hash = table_name(
+            physical_schema=physical_schema,
+            name=name,
+            version=version,
+            catalog=catalog,
+            suffix=suffix,
+            naming_convention=TableNamingConvention.SCHEMA_AND_TABLE,
+        )
+        full_name = f"{c.SQLMESH}_md5__{md5(value_to_hash)}"
+    else:
+        # note: Snapshot._table_name() already strips the catalog from the model name before calling this function
+        # Therefore, a model with 3-part naming like "foo.bar.baz" gets passed as (name="bar.baz", catalog="foo") to this function
+        # This is why there is no TableNamingConvention.CATALOG_AND_SCHEMA_AND_TABLE
+        table_parts = table.parts
+        parts_to_consider = 2 if naming_convention == TableNamingConvention.SCHEMA_AND_TABLE else 1
+
+        # in case the parsed table name has less parts than what the naming convention says we should be considering
+        parts_to_consider = min(len(table_parts), parts_to_consider)
+
+        # bigquery projects usually have "-" in them which is illegal in the table name, so we aggressively prune
+        name = "__".join(sanitize_name(part.name) for part in table_parts[-parts_to_consider:])
+
+        full_name = f"{name}__{version}"
+
     suffix = f"__{suffix}" if suffix else ""
 
-    table.set("this", exp.to_identifier(f"{name}__{version}{suffix}"))
+    table.set("this", exp.to_identifier(f"{full_name}{suffix}"))
     table.set("db", exp.to_identifier(physical_schema))
     if not table.catalog and catalog:
         table.set("catalog", exp.to_identifier(catalog))