Skip to content

Iceberg Catalog Type Mismatch Issue with "drop_data_object_on_type_mismatch" #5392

@HarishRavi96

Description

@HarishRavi96

Issue with Environment : SQLMesh with Iceberg Catalog

https://github.com/TobikoData/sqlmesh/blob/v0.218.0/sqlmesh/core/engine_adapter/base.py

def drop_data_object_on_type_mismatch(
        self, data_object: t.Optional[DataObject], expected_type: DataObjectType
    ) -> bool:
        """Drops a data object if it exists and is not of the expected type.
        Args:
            data_object: The data object to check.
            expected_type: The expected type of the data object.
        Returns:
            True if the data object was dropped, False otherwise.
        """
        if data_object is None or data_object.type == expected_type:
            return False
        logger.warning(
            "Target data object '%s' is a %s and not a %s, dropping it",
            [data_object.to](http://data_object.to/)_table().sql(dialect=self.dialect),
            data_object.type.value,
            expected_type.value,
        )
        self.drop_data_object(data_object)
        return True

I'm encountering an issue with Iceberg catalogs related to the drop_data_object_on_type_mismatch method from PR #5087 #5087. @Iaroslav (Tobiko)

Problem:

In Iceberg catalogs, views appear as "table" objects in information_schema.tables rather than information_schema.views.
This causes:

  • data_object.type returns "table"
  • expected_type is "view"
  • Method drops the physical object before the virtual layer is created

Root Cause:

Iceberg's unified metadata approach treats both views and tables as table-type objects for consistency.

Question:

Should SQLMesh have special handling for Iceberg catalogs where view/table detection works differently, or should the type mismatch logic be more lenient for certain catalog types?

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions