Conversation
izeigerman
force-pushed
the
feat-non-virtual-prod
branch
from
2a32f29 to
afcef50
Compare
izeigerman
force-pushed
the
feat-non-virtual-prod
branch
3 times, most recently
from
b268cea to
c6844d6
Compare
izeigerman
force-pushed
the
feat-non-virtual-prod
branch
from
5e06d2e to
2ac5969
Compare
| DEV_ONLY: Bypass virtual environments in production, using original unversioned model names. | ||
| """ | ||
|
|
||
| FULL = "full" |
There was a problem hiding this comment.
Minor: Is the term "FULL" being used because of "FULL" in auto-categorization? "ALL" feels better to me but I'm wondering if there is some intention behind using "FULL" that I am missing.
| if self.plan.use_finalized_state and not self.virtual_environment_mode.is_full: | ||
| raise ConfigError( | ||
| "Using the finalized state is only supported when `virtual_environment_mode` is set to `full`." | ||
| ) |
There was a problem hiding this comment.
Is this because a "partial" state doesn't make sense if you aren't using VDE in prod? If so, what would be the negative in allowing this? Just doesn't have much meaning?
There was a problem hiding this comment.
Actually, since dev-only VDE always assumes a forward-only plan, this check is no longer relevant. Will remove it, thanks
There was a problem hiding this comment.
I thought about this more and concluded that indeed this configuration is meaningless when used with dev-only VDEs since there are no previous table versions to go back to.
| dev_table_suffix: str | ||
| model_gateway: t.Optional[str] = None | ||
| forward_only: bool = False | ||
| table_naming_convention: TableNamingConvention = TableNamingConvention.default |
There was a problem hiding this comment.
Interesting, I struggled to get this to work. mypy wouldnt let me have it both here and on SnapshotInfoMixin
sungchun12
left a comment
sungchun12
left a comment
There was a problem hiding this comment.
Quick docs suggestions.
izeigerman
force-pushed
the
feat-non-virtual-prod
branch
from
75b9b48 to
282cfe3
Compare
For anyone finding this at a later date, the rebuild required is a manual one, not automatic, at least in 0.210.0 and some projects (in my case multi-repo projects where models reference one-way across projects). When changing virtual_environment_mode from full to dev_only, sqlmesh drops all of the old views first, and does not create the replacement models in the correct order. When it goes to create models further down the chain, they will fail because the model tables it depends on do not exist yet. In that case you can destroy and plan from scratch to make it work. So, when changing a project's virtual_environment_mode from full to dev_only you should be prepared to destroy and plan. |
5 participants
This PR adds support for the
dev_onlyVirtual Data Environment (VDE) mode.In this mode VDE is applied only in development environments, while in production, model tables and views are updated directly, bypassing the virtual layer. This also means that physical tables in production will be created using the original, unversioned model names. Users will continue to benefit from VDE and data reuse across development environments.
The new mode can be enabled in the project configuration:
Switching the mode for an existing project will result in a complete rebuild of all models in the project.