[GLUTEN-4889][VL] feat: Support approx_percentile aggregate function#11651
[GLUTEN-4889][VL] feat: Support approx_percentile aggregate function#11651Yizhou-Yang wants to merge 34 commits intoapache:mainfrom
Conversation
|
Run Gluten Clickhouse CI on x86 |
Yizhou-Yang
changed the title
|
Run Gluten Clickhouse CI on x86 |
|
Please update get-velox.sh to test your PR, then you can verify if both can work well, you may update this line https://github.com/apache/incubator-gluten/blob/5d3f7145cd7fc258aa10b434ea4ec651bd82c764/ep/build-velox/src/get-velox.sh#L28 |
|
Do we need the config? Usually we offload the function to native by default |
added the 16320 and removed the config |
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
Yizhou-Yang
force-pushed
the
percentile0225
branch
from
61dc2ca to
c76cb84
Compare
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
5 similar comments
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Please update the PR description to describe the KLL Sketch is different so that we handle fallback separately. |
done~ |
Yizhou-Yang
force-pushed
the
percentile0225
branch
from
1b402de to
b0679ad
Compare
|
Run Gluten Clickhouse CI on x86 |
…ministic across Spark and Velox implementations
…rom workspace 588c19f24)
…sensitive tests 1. VeloxApproxPercentile: Fix accuracyBuf type from IntegerType to DoubleType to match Velox intermediate type (row(..., double, ...)). 2. VeloxApproxPercentile: Compute K dynamically from accuracy using Velox's kFromEpsilon formula instead of hardcoded DEFAULT_K=200. This ensures the Spark fallback path has the same precision as Velox native execution. 3. GlutenApproximatePercentileQuerySuite: Override precision-sensitive tests with tolerance-based assertions. KLL sketch and GK algorithm inherently select different values at percentile boundaries (e.g., for 1..1000, exact 25th percentile=250.25, GK returns 250 while KLL returns 251). This difference cannot be eliminated by increasing precision.
Yizhou-Yang
force-pushed
the
percentile0225
branch
from
7d22032 to
2c6447e
Compare
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
Yizhou-Yang
force-pushed
the
percentile0225
branch
from
e9a5d18 to
48c34b5
Compare
|
Run Gluten Clickhouse CI on x86 |
3 participants
What
Add Velox
approx_percentilesupport for Spark.Why
Velox uses KLL sketch while Spark uses GK algorithm — their intermediate data formats are incompatible (KLL: 9-field StructType vs GK: single BinaryType buffer). This means fallback between Velox and Spark requires separate handling.
How
VeloxApproximatePercentile: ADeclarativeAggregatewith 9aggBufferAttributesmatching Velox's KLL sketch layout.KllSketchHelper/KllSketchAdd/KllSketchMerge/KllSketchEval): Simplified KLL operations for fallback, binary-compatible with Velox's C++ accumulator.ApproxPercentileRewriteRule: Rewrites Spark'sApproximatePercentileto the Velox-compatible version.Key decisions
IntegerType(Spark's original value); Velox computesepsilon = 1.0/accuracyinternally.Velox dependency
facebookincubator/velox#16320
Related issue: #4889