Skip to content

Add roaring64 BSI comparison and batch value APIs#537

Open
guymolinari wants to merge 3 commits into
masterfrom
bsi-compare-bsi
Open

Add roaring64 BSI comparison and batch value APIs#537
guymolinari wants to merge 3 commits into
masterfrom
bsi-compare-bsi

Conversation

@guymolinari

@guymolinari guymolinari commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds focused roaring64 BSI APIs and optimizations for comparison-heavy workloads. It keeps existing public method signatures intact while adding faster paths for BSI-to-BSI comparisons, aligned batch value extraction, and fused batch equality/value extraction.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Performance improvement
  • Code refactoring
  • Documentation update
  • Test improvements
  • Build/CI changes

Changes Made

What was changed?

  • Added roaring64 BSI-to-BSI comparison support for same-column comparisons.
  • Added aligned batch value extraction so callers can retrieve many BSI values without repeated row-by-row lookups.
  • Added BatchEqualValues, which returns matched column IDs with their matched int64 values in one pass, optionally restricted by a found set.
  • Refactored the optimized int64-width BatchEqual path to share the same trie/cube matching logic used by BatchEqualValues.
  • Preserved generic fallback behavior for wider-than-int64 BSI values.
  • Added correctness tests for signed values, found sets, duplicate query values, big-width fallback, and result isolation.
  • Added benchmark notes in roaring64/BSI_BENCHMARKS.md.

Why?

These APIs avoid common row-by-row GetValue / GetBigValue loops in BSI workloads. In downstream query-engine validation, the fused BatchEqualValues path removed a second lookup pass from relationship-vector reduction and reduced a representative join reduction case from roughly the 480ms band to roughly 288ms median.

Validation

  • go test ./roaring64 -count=1
  • go test ./... -count=1
  • go test ./roaring64 -run '^$' -bench 'BenchmarkBSI64BatchEqual.*LargeFixture' -benchmem -benchtime=2s -count=5
  • go test ./roaring64 -run '^$' -bench 'BenchmarkBSI64CompareBSISameRow' -benchmem -count=5

Downstream smoke validation with a local workspace override:

  • GOWORK=/tmp/qs-roaring.work go test ./qsruntime -count=1
  • Standard TPC-H query suite passed against QuantaStream.
  • Representative q5.060_combined_graph_regional_count median improved to about 288ms from the prior ~480ms band.

Representative Results

See roaring64/BSI_BENCHMARKS.md for benchmark commands and fuller notes.

On the large batch-equality fixture:

  • BatchEqualValues: about 1.6-2.3ms/op
  • BatchEqual + GetBigValues: about 5.4-7.1ms/op
  • BatchEqual + GetValue loop: about 10.9-13.0ms/op

Compatibility

  • Existing public method signatures are unchanged.
  • New APIs are additive.
  • Wider-than-int64 BSI values continue to use generic fallback paths.
  • BatchEqualValues result order is intentionally unspecified.

@guymolinari guymolinari changed the title Add roaring64 BSI-to-BSI comparison Add roaring64 BSI comparison and faster batch value extraction Jul 21, 2026
@guymolinari guymolinari changed the title Add roaring64 BSI comparison and faster batch value extraction Add roaring64 BSI comparison and batch value extraction Jul 21, 2026
@guymolinari guymolinari changed the title Add roaring64 BSI comparison and batch value extraction Add roaring64 BSI comparison and batch value APIs Jul 22, 2026
@guymolinari

Copy link
Copy Markdown
Contributor Author

@lemire - Are you good with this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants