perf: reduce cached long decoding cost - #5869
Draft
peterxcli wants to merge 6 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related to #5485. Depends on #5859.
Review the decoding-only diff (two commits, five files). This PR targets
main, so GitHub also shows the parent commits until #5859 merges; it should be rebased afterward.Rationale for this change
After #5859 removes intermediate row conversion, compressed numeric cache decoding remains expensive. Bypassing compression reduced six-long executor CPU from 494 to 172 ms, identifying the compression stream as the main remaining cost. Spark already uses delta encoding for these numeric values.
What changes are included in this PR?
Store suitable plain long columns as deltas before compression and restore their values in the shared cache reader. Keep validity bits and logical statistics unchanged. Require the delta stream to be over 25% smaller, skip a second compression for irregular full-width longs, and reuse codec factories per partition.
How are these changes tested?
Cache, row-reader, and Kryo suites pass on Spark 3.4.3, 3.5.9, and 4.1.3, including nulls, wrapping long arithmetic, native/Spark readers, projections, and recaching. Splitting the PR preserves the tested source exactly at
2e65a02cb.Benchmark
Spark 4.1.3, Java 17.0.20, AWS AMD EPYC 7282,
local[1], 6 GiB heap, 5M rows, 16 partitions,MEMORY_ONLY, LZ4. Mixed has three longs and three strings; numeric has six sequential/modulo-derived longs. All consuming operators run on Spark.Medians of 40 reads per cell across two fresh JVMs, after five warmups per query. Cache creation and planning are excluded. Parent, Spark, and child results come from separate rounds on the same host; these are descriptive comparisons across those runs. Parent code matches
091eb0020; child is2e65a02cb. Answers match across all nine cases.Whiskers show JVM median ranges. Mixed six-column child medians varied from 986–1,336 ms, and its pooled median was higher than the parent. Count-only and string-only differences do not establish benefits from long-delta decoding. These synthetic values favor delta encoding; native-consumer performance and broader data distributions remain unmeasured.
Repeated cache builds increased from 1.62 to 2.26 s (+39%) for numeric data and 1.49 to 1.61 s (+9%) for random longs versus #5859. Numeric payload shrank from 107.7 to 2.1 MB; random payload remained 241.3 MB. Build figures use six measurements after two warmups per JVM, with reversed case order. Writing needs a temporary Arrow vector and can hold both compressed representations, so low-reuse caches may lose overall.