test: add v9 test tables and test cases

[xushiyan]

Description

• Add v9 COW sample tables with simple/complex keygen, partitioned/non-partitioned, with/without MDT
• Add shared verification logic in hudi-test::v9_verification used by both Rust and Python tests
• Support table version 9 in validation and timeline layout defaulting
• Fix MOR log scanner to correctly handle delete blocks
• Migrate Python test infrastructure from local zip fixtures to Rust test crate via pyo3 testing feature flag
• Add plan_tests for verifying DataFusion explain plans (file pruning, partition pruning)

Closes #301

How are the changes test-covered

• N/A
• Automated tests (unit and/or integration tests)
• Manual tests
  • Details are described below

[Copilot]

Pull request overview

Adds end-to-end test fixtures and verification coverage for Hudi table version 9 across Rust (DataFusion integration) and Python bindings/tests, while updating core table-version validation and timeline defaults to accept v9.

Changes:

• Add v9 txn sample tables (COW + MOR, partitioned + non-partitioned, meta + no-meta) and shared verification helpers.
• Expand Rust integration tests (DataFusion query verification) and update Python tests to use the new test-table plumbing.
• Update core validation/timeline defaults to support table version 9 and adjust log scanner content-type detection for delete blocks.

Reviewed changes

Copilot reviewed 32 out of 46 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
python/tests/test_table_read.py	Updates snapshot/partition/time-travel assertions to use the v8 trips test table fixture.
python/tests/test_table_incremental_read.py	Adds an incremental-read test against a v9 COW txns table using internal test-table path helper.
python/tests/test_table_builder.py	Updates builder-based snapshot read test to use the v8 trips fixture and new expected results.
python/tests/test_file_group_read.py	Updates file-group reader tests to derive a base-file path from the v8 trips table instead of a hardcoded filename.
python/tests/test_datafusion_read.py	Replaces a simple registry check with parametrized v9 DataFusion query verification via a shared helper.
python/tests/conftest.py	Switches from zip-extraction fixture to get_test_table_path-based fixture for v8 trips.
python/src/testing_internal.rs	Introduces PyO3-exposed helpers for locating embedded test tables and verifying v9 txns tables.
python/src/lib.rs	Registers the new testing helpers in the _internal module behind a Cargo feature.
python/Cargo.toml	Adds optional dependency on hudi-test and a testing feature for Python test helpers.
crates/test/src/v9_verification.rs	Adds shared v9 txns table verification logic (DataFusion queries + expected record state).
crates/test/src/lib.rs	Extends SampleTable enum to include v9 tables; removes MDT-specific enum; wires v9 zip existence checks.
crates/test/data/sample_table_use_mdt_basic_schema/mor/avro/v8_nonpartitioned.sql	Removes the MDT v8 non-partitioned SQL fixture content.
crates/test/data/sample_table/mor/avro/v9_txns_simple_nometa.sql	Adds SQL spec for generating v9 MOR simple key, metadata disabled table.
crates/test/data/sample_table/mor/avro/v9_txns_simple_meta.sql	Adds SQL spec for generating v9 MOR simple key, metadata enabled table.
crates/test/data/sample_table/mor/avro/v9_txns_nonpart_nometa.sql	Adds SQL spec for generating v9 MOR non-partitioned, metadata disabled table.
crates/test/data/sample_table/mor/avro/v9_txns_nonpart_meta.sql	Adds SQL spec for generating v9 MOR non-partitioned, metadata enabled table.
crates/test/data/sample_table/mor/avro/v9_txns_complex_nometa.sql	Adds SQL spec for generating v9 MOR complex key, metadata disabled table.
crates/test/data/sample_table/mor/avro/v9_txns_complex_meta.sql	Adds SQL spec for generating v9 MOR complex key, metadata enabled table.
crates/test/data/sample_table/cow/v9_txns_simple_nometa.sql	Adds SQL spec for generating v9 COW simple key, metadata disabled table.
crates/test/data/sample_table/cow/v9_txns_simple_meta.sql	Adds SQL spec for generating v9 COW simple key, metadata enabled table.
crates/test/data/sample_table/cow/v9_txns_nonpart_nometa.sql	Adds SQL spec for generating v9 COW non-partitioned, metadata disabled table.
crates/test/data/sample_table/cow/v9_txns_nonpart_meta.sql	Adds SQL spec for generating v9 COW non-partitioned, metadata enabled table.
crates/test/data/sample_table/cow/v9_txns_complex_nometa.sql	Adds SQL spec for generating v9 COW complex key, metadata disabled table.
crates/test/data/sample_table/cow/v9_txns_complex_meta.sql	Adds SQL spec for generating v9 COW complex key, metadata enabled table.
crates/test/Cargo.toml	Adds an optional datafusion feature for the test crate (DataFusion + tokio deps).
crates/datafusion/tests/query_tests.rs	Adds DataFusion query verification integration tests for all v9 txns table variants.
crates/datafusion/tests/plan_tests.rs	Clarifies test-module intent (plan vs E2E) and adjusts comments/headers.
crates/datafusion/Cargo.toml	Enables hudi-test’s datafusion feature in dev-dependencies to support new query tests.
crates/core/tests/table_read_tests.rs	Updates MDT-enabled table tests to use a v9 MDT-enabled sample table.
crates/core/src/timeline/builder.rs	Defaults table version >= 8 to timeline layout v2 and adds a v9 defaulting test.
crates/core/src/table/validation.rs	Adds v9 to supported table versions and adds a validation test for v9+layout2.
crates/core/src/file_group/log_file/scanner.rs	Adjusts content-type detection so delete blocks don’t trigger “mixed type” errors.

[codecov]

Codecov Report

❌ Patch coverage is 75.78947% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.91%. Comparing base (4df94b9) to head (7e9f74f).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
python/src/testing_internal.rs	0.00%	16 Missing ⚠️
crates/test/src/v9_verification.rs	89.39%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #536      +/-   ##
==========================================
- Coverage   85.01%   84.91%   -0.11%     
==========================================
  Files          71       73       +2     
  Lines        4530     4606      +76     
==========================================
+ Hits         3851     3911      +60     
- Misses        679      695      +16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.