feat(operator): cut a numeric column into bins - #8481
Draft
kz930 wants to merge 3 commits into
Draft
Conversation
27 tasks
kz930
marked this pull request as draft
September 10, 2026 01:31
Contributor
Automated Reviewer SuggestionsBased on the
|
github-actions
Bot
added
feature
dependencies
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8481 +/- ##
============================================
- Coverage 93.62% 93.62% -0.01%
- Complexity 4857 4873 +16
============================================
Files 1212 1214 +2
Lines 50037 50079 +42
Branches 6132 6136 +4
============================================
+ Hits 46847 46886 +39
- Misses 1676 1677 +1
- Partials 1514 1516 +2
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 378 | 0.231 | 25,772/33,887/33,887 us | 🟢 -19.7% / 🔴 +110.0% |
| 🔴 | bs=100 sw=10 sl=64 | 781 | 0.476 | 122,309/167,811/167,811 us | 🔴 +17.8% / 🔴 +54.4% |
| ⚪ | bs=1000 sw=10 sl=64 | 907 | 0.553 | 1,103,357/1,163,102/1,163,102 us | ⚪ within ±5% / 🔴 +12.1% |
Baseline details
Latest main 75c85aa from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 378 tuples/sec | 400 tuples/sec | 755.28 tuples/sec | -5.5% | -50.0% |
| bs=10 sw=10 sl=64 | MB/s | 0.231 MB/s | 0.244 MB/s | 0.461 MB/s | -5.3% | -49.9% |
| bs=10 sw=10 sl=64 | p50 | 25,772 us | 22,790 us | 12,957 us | +13.1% | +98.9% |
| bs=10 sw=10 sl=64 | p95 | 33,887 us | 42,216 us | 16,134 us | -19.7% | +110.0% |
| bs=10 sw=10 sl=64 | p99 | 33,887 us | 42,216 us | 20,333 us | -19.7% | +66.7% |
| bs=100 sw=10 sl=64 | throughput | 781 tuples/sec | 808 tuples/sec | 980.1 tuples/sec | -3.3% | -20.3% |
| bs=100 sw=10 sl=64 | MB/s | 0.476 MB/s | 0.493 MB/s | 0.598 MB/s | -3.4% | -20.4% |
| bs=100 sw=10 sl=64 | p50 | 122,309 us | 120,351 us | 101,894 us | +1.6% | +20.0% |
| bs=100 sw=10 sl=64 | p95 | 167,811 us | 142,426 us | 108,718 us | +17.8% | +54.4% |
| bs=100 sw=10 sl=64 | p99 | 167,811 us | 142,426 us | 122,482 us | +17.8% | +37.0% |
| bs=1000 sw=10 sl=64 | throughput | 907 tuples/sec | 914 tuples/sec | 1,011 tuples/sec | -0.8% | -10.3% |
| bs=1000 sw=10 sl=64 | MB/s | 0.553 MB/s | 0.558 MB/s | 0.617 MB/s | -0.9% | -10.4% |
| bs=1000 sw=10 sl=64 | p50 | 1,103,357 us | 1,092,112 us | 996,422 us | +1.0% | +10.7% |
| bs=1000 sw=10 sl=64 | p95 | 1,163,102 us | 1,155,308 us | 1,037,670 us | +0.7% | +12.1% |
| bs=1000 sw=10 sl=64 | p99 | 1,163,102 us | 1,155,308 us | 1,072,152 us | +0.7% | +8.5% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,529.35,200,128000,378,0.231,25772.28,33886.74,33886.74
1,100,10,64,20,2562.45,2000,1280000,781,0.476,122308.90,167811.31,167811.31
2,1000,10,64,20,22056.41,20000,12800000,907,0.553,1103357.35,1163102.20,1163102.20
carloea2
reviewed
Sep 10, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
Additional boundary cases reproduced with compiled local tests.
The same gap as the timestamp parts, on the other kind of column. Grouping by a continuous number puts almost every row in a group of its own, so counts per age bracket or per price band could not be asked for; a user wanting them had to write a Python UDF. Two cuts, because they answer different questions. Equal width divides the span into bins of one size; equal frequency cuts at the quantiles, so the bins hold about as many rows as each other. The second is why the operator reads the whole table before it emits: a quantile is not known until the last row has arrived. The bin reads as its own range, "(2.5, 5.0]", which is a label to group by rather than a number to do arithmetic on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
removed
dependencies
An equal-width cut has no edges to find when every cell of the column is empty, and none when no row arrived at all, so pandas raises before the suffix that keeps a hole a hole can run. Both renderings now skip the cut in that case and let the suffix turn the holes into the empty bin labels they already are. The equal-frequency cut does not raise on the same table, but it goes through the same guard: one shape of answer for a table with nothing in it, whichever way it was asked to cut. Two tests run the exported script over an all-empty column, an empty table and a column with one hole, since the defect is what pandas does at runtime rather than what the operator writes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
@carloea2 the finding is fixed: a column with nothing in it skips the cut, so the rows keep null bin labels. Would you take another look when you have a moment? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
An operator that cuts a numeric column into bins and adds the bin a row fell in as a column of its own.
Two cuts, because they answer different questions. Equal width divides the span into bins of one size; equal frequency cuts at the quantiles, so the bins hold about as many rows as each other. The second is why the operator reads the whole table before it emits: a quantile is not known until the last row has arrived.
The bin reads as its own range,
(2.5, 5.0], which is a label to group by rather than a number to do arithmetic on.Any related issues, documentation, discussions?
Not part of #8325: that set makes the operators Texera already had exportable. This adds a new one, which implements the trait from the start rather than gaining it. The trait is in #8327, which has merged.
Closes #8479, the task this change is the whole of.
The same gap as #8478, on the other kind of column. Grouping by a continuous number puts almost every row in a group of its own, so counts per age bracket or per price band could not be asked for; a user wanting them had to write a Python UDF.
How was this PR tested?
Ten unit tests in the operator's own spec, over the schema it declares and the Python it emits on both paths. The parity harness runs it on four configurations, one per cut plus a hostile column name and an empty cell, comparing the engine's answer against the exported script's.
The operator in the editor, cutting
scoreinto four equal-width bins:Each row's bin beside the value it came from, named by the range it fell in rather than by a number:
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)