Skip to content

Remove unsupported dataType cases in VeloxBackend#11767

Open
VvanFalleaves wants to merge 2 commits intoapache:mainfrom
VvanFalleaves:main
Open

Remove unsupported dataType cases in VeloxBackend#11767
VvanFalleaves wants to merge 2 commits intoapache:mainfrom
VvanFalleaves:main

Conversation

@VvanFalleaves
Copy link

Removed handling for StructType, ArrayType, and MapType in dataType matching.

DROP TABLE IF EXISTS hive_support;
CREATE TABLE IF NOT EXISTS hive_support (
  int_col INT,
  array_col ARRAY<INT>,
  map_col MAP<STRING, INT>,
  struct_col STRUCT<field1: INT, field2: STRING, field3: BOOLEAN>
)
PARTITIONED BY (part_col INT)
STORED AS PARQUET
TBLPROPERTIES (
  'parquet.compression' = 'ZSTD',
  'parquet.compression.zstd.level' = '5',
  'parquet.enable.dictionary' = 'false'
);
WITH number_seq AS (
  SELECT CAST(id AS INT) AS id_int
  FROM range(1, 50000000, 1, 100)
)
INSERT OVERWRITE TABLE hive_support
PARTITION (part_col) 
SELECT
  id_int AS int_col,
  ARRAY(
    id_int % 10,
    id_int % 100,
    id_int % 1000
  ) AS array_col,
  MAP(
    CONCAT('key_', CAST(id_int % 5 AS STRING)), id_int % 100,
    CONCAT('key_', CAST((id_int + 1) % 5 AS STRING)), (id_int + 1) % 100
  ) AS map_col,
  named_struct(
    'field1', id_int,
    'field2', CONCAT('str_', CAST(id_int AS STRING)),
    'field3', id_int % 2 = 0
  ) AS struct_col,
  id_int % 10 AS part_col
FROM number_seq
SORT BY id_int;

The following error is reported during the execution:

26/03/10 20:50:46 WARN GlutenFallbackReporter: Validation failed for plan: WriteFiles[QueryId=4], due to: Unsupported native write: Found unsupported type:ArrayType,MapType,StructType.

The Velox version downloaded by gluten in the branch-1.3 is https://github.com/oap-project/velox/tree/gluten-1.3.0. In this case, HiveDataSink::appendData does not support PARTITIONED BY and complex types. However, in newer Velox, HiveDataSink::appendData method has been updated, but gluten does not fix it.

Removed handling for StructType, ArrayType, and MapType in dataType matching.
@github-actions github-actions bot added the VELOX label Mar 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant