[VL] Gluten performs over 10x slower than Vanilla Spark on simple "select ... limit ..." queries

[zhixingheyi-tian]

Description

Simple case as below

select * from store_sales limit 10;

Spark behaviour

Vanilla Spark execution is very fast , and generates only one task.

Gluten behaviour

Gluten performs over 10x slower than Vanilla Spark. And generates multiple tasks.

@weiting-chen @zhouyuan @jinchengchenghh

Thanks

[zhouyuan]

@zhixingheyi-tian Hi, this seems to be a known issue, the root case is Gluten needs to read 4096 rows at least. Using smaller batch size spark.gluten.sql.columnar.maxRecordsPerBatch may help

[wForget]

This is because gluten executes the entire RDD, whereas vanilla spark only executes a single partition of one RDD. This may be because ColumnarCollectLimitExec does not override the executeCollect method.

[FelixYBW]

Collect operator is a special one in Gluten, I remember @JkSelf noted it before.

[JkSelf]

@FelixYBW It seems the root cause is that ColumnarCollectLimitExec in Gluten does not override the executeCollect method. This causes Gluten to process the entire dataset instead of the limited subset as expected.

@wForget Could you help to fix this issue? If not, I'm happy to take this over and submit a PR. Thanks!