Spark 4.1: Migrate to new version framework in DSv2#15240
Spark 4.1: Migrate to new version framework in DSv2#15240aokolnychyi wants to merge 1 commit intoapache:mainfrom
Conversation
|
This PR is NOT ready for review just yet. |
|
@aokolnychyi, thanks for the PR. Do we really call it version framework in Spark? I'm not able to find any documentation. |
|
@manuzhang, that's how I tend to call it, not official by any means :) The core idea is that Spark has a notion of versioned tables and keeps track of the versions. |
aokolnychyi
force-pushed
the
spark-dsv2-version-framework-rebased
branch
from
558e336 to
748cbec
Compare
aokolnychyi
changed the title
| TableProperties.MERGE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName()); | ||
| } | ||
|
|
||
| @TestTemplate |
There was a problem hiding this comment.
This test was verifying a particular limitation that we couldn't concurrently modify the table from another thread. This is no longer an issue because we pin the scanned snapshot so it doesn't matter if the underlying table evolves in the meantime.
| * under the License. | ||
| */ | ||
| package org.apache.iceberg.spark.sql; | ||
| package org.apache.iceberg.spark.extensions; |
There was a problem hiding this comment.
Branching will require extensions in 4.1 because we must pin the branch during analysis. Spark doesn't let us do this cleanly today (see ResolveBranch). It will be fixed in 4.2.
aokolnychyi
force-pushed
the
spark-dsv2-version-framework-rebased
branch
3 times, most recently
from
7de899b to
10fd253
Compare
| TableProperties.DELETE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName()); | ||
| } | ||
|
|
||
| @TestTemplate |
There was a problem hiding this comment.
Same explanation as in MERGE below.
| } | ||
|
|
||
| @TestTemplate | ||
| public synchronized void testUpdateWithConcurrentTableRefresh() throws Exception { |
There was a problem hiding this comment.
Same explanation as in MERGE.
aokolnychyi
force-pushed
the
spark-dsv2-version-framework-rebased
branch
from
10fd253 to
c5a30b9
Compare
3 participants
This PR migrates our Spark 4.1 connector to new version framework in DSv2.