Feat(dbt): Add support for dbt custom materializations

[themisvaltinos]

This update adds support for dbt custom materializations which allows users to define their own strategies via jinja macros.

Docs: https://docs.getdbt.com/guides/create-new-materializations?step=2

[izeigerman]

what is this adapter?

[themisvaltinos]

it's to allow users to define different materialisation tied to different adapters and for dispatch then to appropriately match the correct one used

{% materialization my_materialization_name, default %}
 -- cross-adapter materialization... assume Redshift is not supported
{% endmaterialization %}

{% materialization my_materialization_name, adapter='redshift' %}
-- override the materialization for Redshift
{% endmaterialization %}

test for this example: test_adapter_specific_materialization_override

[izeigerman]

What is the point of this override?

[themisvaltinos]

thank you good catch. I had originally the definition there and then realised it should be in the data hash and moved it but forgot to remove the metadata_hash_values entirely since it's not needed will do

[izeigerman]

Does the custom materialization strategy calls the pre-hooks?

[themisvaltinos]

yes internally using run_hooks so it's up to the user when they should call them or if the specific materialisation doesn't call them at all: https://docs.getdbt.com/guides/create-new-materializations?step=2#run-pre-hooks

[izeigerman]

ok, can we instead extend the materialization strategy with:

    def run_pre_statements(...)
    
    def run_post_statements(...)

and in the default implementation we do

adapter.execute(model.render_pre_statements(**render_statements_kwargs))

while for DbtCustomMaterializationStrategy it's a noop

[themisvaltinos]

yes that seems better, revised it with run methods instead in the materialisation strategy

[izeigerman]

why do we need to use getattr instead of accessing the attribute directly?

[themisvaltinos]

yes good point I forgot all our model classes have jinja_macros

[izeigerman]

Why do we manually construct the context instead of relying on JinjaMacros to do it for us?

[themisvaltinos]

so that I didn't use its private method, but I revised it to use jinjamacros instead if this is what you meant. if you meant to do it through build_environment that got quite complicated and it updates internally the macros that were needed which lead to failures. if you mean why this entire logic is needed in the first place it is because we need to inject runtime values specific to the materialization, like the actual sql of the query being materialized, the pre/post hooks, the physical table name parts if it’s a vde etc.

[izeigerman]

Let's follow the existing naming convention and add Strategy at the end of the name