feat(workflow-operator): export the relational and sampling operators - #8506
Draft
kz930 wants to merge 1 commit into
Draft
feat(workflow-operator): export the relational and sampling operators#8506kz930 wants to merge 1 commit into
kz930 wants to merge 1 commit into
Conversation
Seventeen operators implement the trait: the three joins, the three set operations, Aggregate, the sort family, Split, the two samplers, and If, Dummy and Sleep. What they have in common is that they rearrange rows rather than read what is inside one. A sampler decides per row whether to keep it, so which rows survive is fixed by the exact sequence java.util.Random produces. Seeding Python's own generator selects a different set, and the script would then report a different sample than the workflow it came from, so SamplingHelpers transcribes the generator rather than approximating it. Split out of apache#8340 on review, which had grown past what one reading can hold. That change keeps the operators that work on text, and the cast transcription they share. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Automated Reviewer SuggestionsBased on the
|
Contributor
📊 Arrow Flight E2E bench(no arrow-flight-e2e.csv in artifact) |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8506 +/- ##
============================================
- Coverage 93.69% 93.69% -0.01%
+ Complexity 4826 4810 -16
============================================
Files 1209 1209
Lines 49871 49855 -16
Branches 6099 6099
============================================
- Hits 46727 46711 -16
Misses 1652 1652
Partials 1492 1492
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
27 tasks
kz930
marked this pull request as draft
September 11, 2026 05:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Sixteen operators implement
StandaloneCodeGenerator: the three joins, the three set operations, Aggregate, the sort family, Split, the two samplers, and If, Dummy and Sleep. What they have in common is that they rearrange rows rather than read what is inside one.The samplers need more than a pandas expression. A sampler decides per row whether to keep it, so which rows survive is fixed by the exact sequence
java.util.Randomproduces; seeding Python's own generator with the same number selects a different set, and the script would then report a different sample than the workflow it came from.SamplingHelperstranscribes the generator, and the three operators that draw from one emit it once per script.#8340 keeps the operators that work on text, and the cast transcription they share.
Any related issues, documentation, discussions?
Part of #8325, 11 of 27; that issue lists the set in order.
Closes #8505, the task this change is the whole of.
How was this PR tested?
Each operator asserts the block it emits in its own spec. The joins carry the cases a review found: a hash join whose left frame already holds the probe key's name drops the renamed right key rather than the left payload, and an interval join reads its keys off the merged frame rather than writing over an input column that happens to share a temporary's name.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
🤖 Generated with Claude Code