Skip to content

bench: add ASOF join benchmarks - #23833

Open
Xuanwo wants to merge 10 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-benchmarks
Open

bench: add ASOF join benchmarks#23833
Xuanwo wants to merge 10 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-benchmarks

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Jul 23, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

This is the final core benchmark layer of the ASOF JOIN stack. It uses the
standard SQL benchmark framework so the workloads share the existing runner,
result format, and Criterion mode.

What changes are included in this PR?

  • Add an asof_join SQL benchmark suite, runnable with
    ./bench.sh run asof_join or benchmark_runner.
  • Add seven workloads covering relative input sizes, available ordering,
    equality-group cardinality and skew, match direction, and payload width.
  • Describe each workload and its explicit ASOF match semantics in the query.
  • Generate pre-sorted Parquet inputs for one keyed workload in its load hook and
    declare their ordering with WITH ORDER, keeping input sorting outside that
    measured query.
  • Assert that every workload plans to AsOfJoinExec.

Are these changes tested?

Yes:

  • cargo fmt --all
  • ./ci/scripts/doc_prettier_check.sh --write --allow-dirty
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test -p datafusion-benchmarks --lib --bins
  • cargo run -p datafusion-benchmarks --bin benchmark_runner -- asof_join --iterations 1 --partitions 4
  • CARGO_COMMAND='cargo run' ./bench.sh run asof_join 1
  • cargo run -p datafusion-benchmarks --bin benchmark_runner -- asof_join --query 7 --partitions 4 --criterion

The pre-sorted workload's physical plan contains AsOfJoinExec without an
input SortExec. These smoke runs validate the suite and are not presented as
performance claims.

Are there any user-facing changes?

This adds ASOF workloads to the repository benchmark tooling. It does not
change query semantics or runtime behavior.

This is the final core item tracked by #23738. It does not depend on the
optional floating-point follow-up #24375.

@github-actions github-actions Bot added documentation Improvements or additions to documentation sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 23, 2026
@codecov-commenter

codecov-commenter commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.88%. Comparing base (681705e) to head (f0690ba).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23833      +/-   ##
==========================================
- Coverage   81.88%   81.88%   -0.01%     
==========================================
  Files        1133     1133              
  Lines      424522   424522              
  Branches   424522   424522              
==========================================
- Hits       347622   347612      -10     
- Misses      56288    56294       +6     
- Partials    20612    20616       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 23, 2026
@github-actions github-actions Bot added the physical-expr Changes to the physical-expr crates label Aug 14, 2026
@Xuanwo
Xuanwo force-pushed the xuanwo/asof-benchmarks branch from e826548 to 1ff6d2b Compare August 20, 2026 17:56
@Xuanwo
Xuanwo force-pushed the xuanwo/asof-benchmarks branch from 1ff6d2b to c3d099b Compare August 28, 2026 05:40
@github-actions github-actions Bot removed the physical-expr Changes to the physical-expr crates label Aug 28, 2026
Xuanwo added 4 commits August 30, 2026 21:47
# Conflicts:
#	datafusion/expr/src/logical_plan/builder.rs
#	datafusion/expr/src/logical_plan/plan.rs
#	datafusion/sql/src/relation/join.rs
#	datafusion/sql/src/unparser/plan.rs
#	datafusion/sqllogictest/test_files/asof_join.slt
#	datafusion/substrait/tests/cases/serialize.rs
#	docs/source/user-guide/sql/select.md
@github-actions github-actions Bot removed documentation Improvements or additions to documentation sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Sep 13, 2026
@github-actions github-actions Bot removed substrait Changes to the substrait crate proto Related to proto crate auto detected api change Auto detected API change labels Sep 13, 2026
@Xuanwo
Xuanwo marked this pull request as ready for review September 13, 2026 04:10
@Xuanwo

Xuanwo commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

There are many optimizations to do, but I think it good to have a benchmark in first. This PR will be the last one of this stack, all other changes will be non-blocking, optional follow ups to do.

Cc @2010YOUY01 and @jayzhan211, let's go! Time to close the issue created 5 years ago 🤓

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Xuanwo

Similar to nlj, hj, smj it would be nice to have ./bench.sh run asof

Some suggestions for more cases covered

Comment thread benchmarks/src/asof.rs Outdated
Comment thread datafusion/physical-plan/benches/asof_join.rs Outdated

@2010YOUY01 2010YOUY01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the fast iteration on this project! Here are some suggestions

I suggest we move the SQL benchmark to the new SQL bench framework, like https://github.com/apache/datafusion/tree/main/benchmarks/sql_benchmarks/nlj, so it would be runnable with

# under benchmarks/
./bench.sh run asof_join

Optional: To better explain the workload covered, we can write comments like

# top comment: axes to vary
#
# size: vary left/right input with very small / medium / large sizes, like 100/100k/10m rows
# sortedness: does the input already have the required order, or do we have to sort?
# group cardinality / size: ...

# per-query comment: just tag the axes
Q1: left small, right large; no existing order; single group ...

We can do this as a follow-up and potentially examine and try to better cover the typical workload.

Comment thread benchmarks/src/asof.rs Outdated
Comment thread benchmarks/src/asof.rs Outdated
Comment thread datafusion/physical-plan/benches/asof_join.rs Outdated
@github-actions github-actions Bot removed the physical-plan Changes to the physical-plan crate label Sep 13, 2026
@Xuanwo

Xuanwo commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

Thanks @jayzhan211 and @2010YOUY01 for the thoughtful reviews! This is much cleaner now 🙌

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants