Add roaring64 BSI comparison and batch value APIs#537
Open
guymolinari wants to merge 3 commits into
Open
Conversation
Contributor
Author
|
@lemire - Are you good with this? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds focused roaring64 BSI APIs and optimizations for comparison-heavy workloads. It keeps existing public method signatures intact while adding faster paths for BSI-to-BSI comparisons, aligned batch value extraction, and fused batch equality/value extraction.
Type of Change
Changes Made
What was changed?
BatchEqualValues, which returns matched column IDs with their matched int64 values in one pass, optionally restricted by a found set.BatchEqualpath to share the same trie/cube matching logic used byBatchEqualValues.roaring64/BSI_BENCHMARKS.md.Why?
These APIs avoid common row-by-row
GetValue/GetBigValueloops in BSI workloads. In downstream query-engine validation, the fusedBatchEqualValuespath removed a second lookup pass from relationship-vector reduction and reduced a representative join reduction case from roughly the 480ms band to roughly 288ms median.Validation
go test ./roaring64 -count=1go test ./... -count=1go test ./roaring64 -run '^$' -bench 'BenchmarkBSI64BatchEqual.*LargeFixture' -benchmem -benchtime=2s -count=5go test ./roaring64 -run '^$' -bench 'BenchmarkBSI64CompareBSISameRow' -benchmem -count=5Downstream smoke validation with a local workspace override:
GOWORK=/tmp/qs-roaring.work go test ./qsruntime -count=1q5.060_combined_graph_regional_countmedian improved to about 288ms from the prior ~480ms band.Representative Results
See
roaring64/BSI_BENCHMARKS.mdfor benchmark commands and fuller notes.On the large batch-equality fixture:
BatchEqualValues: about 1.6-2.3ms/opBatchEqual + GetBigValues: about 5.4-7.1ms/opBatchEqual + GetValue loop: about 10.9-13.0ms/opCompatibility
BatchEqualValuesresult order is intentionally unspecified.