Feat: Improve CLI and --explain output for restatements

[erindru]

This PR builds on #5285 to make it easier to reason about what's going on with restatements, which are a critical part of seamless dbt compatibility.

Currently, "restatements" has some nuance:

• Many things assume a plan is a restatement plan if plan.restatements is set. However, plan.restatements gets set for dev previews, leading to confusion when SQLMesh outputs "Restating models" when no restatements were specified
• We don't show how the selector for --restate-model maps to the backfill item list, leading to "why do I suddenly have 100 models to backfill when I only specified 3"-type questions
• In --explain, we currently just say "some intervals might cleared", instead of specifically listing the affected snapshots and what environments theyre promoted in

Current state (before this PR)

Dev previews

• notice that it says "Restating models" even though --restate-model was not specified

Actual restatement (with --explain)

• We say "Restating models" without listing anything
• The backfill model list has a model that wasnt even specified - why is erin.test there when I just wanted to restate sqlmesh_example.full_model??
• We do indicate that we invalidate some intervals, but we don't go into exactly what "invalidate" means and what will be affected by that

After this PR

Dev previews

• we simply dont output the "Restating models" indicator, because we arent actually restating any models

Actual restatement (no --explain)

• We list which models matched the selector separately from the models to backfill
• We include an explanation as to why the backfill list may be larger than expected

Actual restatement (with --explain)

• The --explain output makes it clear that only state is affected by this plan
• It explains that only tables in development environments are affected
• It then exactly lists which tables are affected and which environments they're in

[izeigerman]

I honestly don't know if it's worth doing. This will slow down the explainer a lot. Furthermore, users won't even see this by default since the large list will just be collapsed. So we'll be significantly slowing down the whole explain for the sake of something very few people will even look at (if ever).

We should only do this in the super verbose mode. Alternatively, I'm ok with not having this level of detail at all.

[erindru]

The main reason I did this was for consistency with the other stages, which do output the physical table names (eg, the backfill stage). Also, without the physical table names, its difficult to indicate the snapshots that aren't promoted in any environment but are still affected.

This will slow down the explainer a lot

It wont affect the explainer at all unless restatements are both present and affect a lot of non-prod snapshots that are not already present in the local cache.

I did try to come up with a lighter way of generating correct physical table names without loading full Snapshots but I quickly aborted because of the amount of properties that are needed.

Furthermore, users won't even see this by default since the large list will just be collapsed

Ah, fair, I missed the call to _limit_tree.

Personally I don't see the point in limiting explain output ever because a partial explanation that omits output for any entry over the MAX_TREE_LENGTH of 10, especially in --verbose mode, is simply not that useful for debugging.

However, to keep in line with the existing convention, i've switched this to only load snapshots / print the full table names for -vv (Verbosity.VERY_VERBOSE)

[erindru]

Example of default output (without -vv) now:

[izeigerman]

I think the current (default) reporting is fine. I don't think reporting physical tables adds that much value

[erindru]

Ok, i'll wind it back.

But I would like to point out that one of the major criticisms of SQLMesh is that it's hard to tell exactly what's going on, which we don't help if the --explain output is intentionally vague

[izeigerman]

Just to confirm: what you showed on the screenshot above I think is the sufficient amount of detail. Further more, I think it's more useful than individual physical tables. I think having details is important, but having too much details is overwhelming and has the opposite effect.

[erindru]

Understood, i've reduced the scope of this PR accordingly :)