Skip to content

perf: speed up Spark consumers of Comet cache - #5859

Draft
peterxcli wants to merge 4 commits into
apache:mainfrom
peterxcli:codex/cache-spark-consumer-benchmark
Draft

perf: speed up Spark consumers of Comet cache#5859
peterxcli wants to merge 4 commits into
apache:mainfrom
peterxcli:codex/cache-spark-consumer-benchmark

Conversation

@peterxcli

@peterxcli peterxcli commented Sep 11, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Related to #5485.

Numeric cache encoding and decoding is split into #5869, stacked on this PR.

Rationale for this change

Spark can read Comet's cached Arrow vectors through a row iterator even when the consumer supports code generation. This materializes an intermediate UnsafeRow for every row before the consumer reads its fields.

What changes are included in this PR?

Feed eligible cache scans through Spark's ColumnarToRowExec, which fuses vector reads into the generated consumer. Preserve AQE cache stages and existing columnar boundaries. For remaining row readers, generate an indexed iterator using Spark's reusable UnsafeRow writer, removing adapters and the extra copy while preserving owned variable-width values and interpreted fallback.

How are these changes tested?

Cache and iterator suites pass on Spark 3.4.3, 3.5.9, and 4.1.3, covering cold/warm AQE caches, codegen and interpreted paths, row ownership, nulls, nested values, sorting, joins, and batch boundaries.

Benchmark

The patch reduces cached-read time by 42–78% versus Comet main. Reading all six mixed columns takes 28% less time than vanilla Spark; reading six numeric columns still takes 15% more time.

Spark 4.1.3, JDK 21, Apple M4, 6 GiB heap, one local worker; 5M rows and six columns. Mixed uses three longs and three strings; numeric uses six longs. All queries use Spark operators with Comet native execution disabled. Vectorized cache reading is enabled for all three cases: vanilla Spark and Comet main choose row readers; the patch uses the fused columnar path.

Medians of 30 actions per cell across two fresh JVMs, with five warm-ups per query and reversed run order. Cache creation and planning are outside timing. Main: 8320ae481; measured patch: cd80194cd (reader code unchanged at 091eb0020). Patch / Spark is the elapsed-time ratio; lower is better.

Schema Columns read Vanilla Spark (ms) Comet main (ms) Comet patch (ms) Patch / Spark
Mixed count(*) 52.85 148.42 42.64 0.81×
Mixed 1 long 64.22 198.10 66.38 1.03×
Mixed 1 string 173.64 310.53 120.80 0.70×
Mixed 3 columns 326.77 425.84 219.41 0.67×
Mixed 6 columns 503.43 624.37 363.18 0.72×
Numeric count(*) 37.22 135.37 29.30 0.79×
Numeric 1 long 60.21 183.93 53.07 0.88×
Numeric 3 columns 94.06 244.05 101.92 1.08×
Numeric 6 columns 157.03 313.40 180.37 1.15×

Grouped bar chart comparing vanilla Spark cache, Comet main, and Comet patch in milliseconds

A separate forced-row control takes 541 ms for six mixed columns and 226 ms for six numeric columns, versus 363 and 180 ms with the columnar path. All 1,080 measured actions, including this control, matched uncached answers. Spark six-column medians varied from 488–505 ms for mixed and 151–168 ms for numeric between JVMs. These are cached aggregate reads on one machine, not whole-application speedups.

@github-actions github-actions Bot added enhancement New feature or request performance labels Sep 11, 2026
@peterxcli peterxcli changed the title perf: reduce Spark cache row conversion overhead perf: speed up Spark consumers of Comet cache Sep 11, 2026
@peterxcli
peterxcli force-pushed the codex/cache-spark-consumer-benchmark branch 2 times, most recently from 2e65a02 to 091eb00 Compare September 11, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant