feat(workflow-operator): export the source operators as Python by kz930 · Pull Request #8341 · apache/texera

kz930 · 2026-09-02T18:18:44Z

What changes were proposed in this PR?

Nine source operators implement StandaloneCodeGenerator, so an exported script starts from the same data the workflow did rather than from a variable the reader has to fill in: the CSV family, JSON Lines, Arrow, plain text, and the two that read a file named at run time.

A source is the one place where the script cannot simply repeat what the operator does. The engine resolves a dataset through Texera's storage and hands the operator a URI; a script has no such resolver, so it reads the file from its own directory under the name that URI ended with. That is what makes an exported script portable, and it is equally its one precondition: the data has to sit beside the script. The name is taken from the last path segment rather than by parsing the whole string as a URI, because the resolver percent-encodes the file-relative parts but leaves the repository and version names as the user typed them, and a dataset version with a space in its name makes new URI throw before any code is generated.

Two sources are reported as unverifiable rather than exported blind. File Scan takes its filenames from an input port at run time, which a source harness has nothing to feed. URL Fetcher reads a live URL, so two runs are not required to agree and a comparison would only measure the network.

Any related issues, documentation, discussions?

Part of #8325, 12 of 27; that issue lists the set in order. It needs #8327 for the trait, so it does not compile until that lands, and the rows these operators add to the verification runner follow with the harness rather than as whole new files here.

Closes #8415, the task this change is the whole of.

How was this PR tested?

Each operator asserts the block it emits in its own spec. Once the verification lands it is also run through the engine and through its generated script, on every configuration its schema offers, and the two answers compared; this branch is cut from main and does not carry that machinery, so those runs are not on this diff's CI.

Arrow is the one worth calling out. Its column of timestamps only matches once #7672 is in: before that the engine read the file's numbers through the JVM's zone while pd.read_feather read the wall clock the file states, and the two differed by whatever the machine was set to. That change has since landed.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

…ython script A workflow can be read in the editor but not taken away: there is no form of it that runs anywhere else, so a user who wants to keep a pipeline, hand it to someone without Texera, or step through it in a notebook has nothing to take. This adds the seam for one and the first few operators through it. An operator says how it reads outside the engine by implementing `StandaloneCodeGenerator`, returning a block of pandas that names its inputs and outputs as `in1df` / `out1df`. The translator walks the plan in topological order, gives every port a variable, substitutes those placeholders, and prints the leaves; `inAlldf` stands for the whole list of upstreams, which is what a variadic port like Union's needs, since any fixed count the code stated would be wrong for some workflow. An operator with no generator yet leaves a commented TODO rather than a line that looks like it works. `GET /workflow-to-python` on the compiling service returns the script for a plan it is given. Five operators implement it here — Distinct, Limit, Projection, Filter and Union — chosen to cover the shapes the translator has to handle: a single input, a config-driven one, one that renames columns, one that builds a predicate, and the variadic port. The rest of the operator set follows in later changes. `pyStringLiteral` renders a value as a Python literal with the escaping that keeps a quote or a newline in a column name from ending the literal early. The generators cannot use the runtime's decode expression, which needs an operator instance to decode through. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…re the files The standalone export claims that a generated script does what the operator does. Nothing checks it. This adds the two runners that make the claim checkable, and the file format they meet in. `OpExecHarness` runs a LogicalOp the way the engine does — compiling it to a physical plan and driving the executor — but outside a workflow, against JSONL files rather than a live upstream. `PyOpExecHarness` does the same for a Python operator, through the worker the engine uses. `StandaloneRunner` takes the other path: it asks the operator for its standalone code, wraps it in a script that binds `in1df` from the same files, and runs it. `TupleIO` is what the two meet in. A JSONL row carries values and no types, so the schema travels beside it in a sidecar; without one, a column written as INTEGER reads back as a number and the two paths disagree over a difference neither operator made. Both runners produce files, not assertions, so what to make of a difference is left to a later change. What is here is enough to run one operator both ways and see that the answers match, which is what the spec does with Distinct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…he operator it came from The standalone export claims a generated script does what the operator does. This is what checks it, for every operator, on every configuration the operator offers. An operator is run twice. `OpExecHarness` drives it the way the engine does, compiled to a physical plan but outside a workflow, reading JSONL files rather than a live upstream; `PyOpExecHarness` does the same for a Python operator through the worker the engine uses. `StandaloneRunner` takes the other path, wrapping the operator's standalone code in a script that binds the same files. Both write files, and `Comparator` reads them back: order-insensitive by default, since the engine interleaves across workers and only the sort family promises an order. A visualization is compared as a figure rather than as a frame. What to run an operator ON is decided rather than written by hand for each. `ConfigGenerator` reads the operator's own schema — its enums, defaults, declared ranges and column pickers — and produces a base configuration plus one variant per branch the operator offers, so a switch nobody thought to try is still tried. `CanonicalFixture` is the table they run against, one column per shape an operator might ask for. `CuratedHandlers` is the escape hatch for an operator whose input cannot be derived, and `TransformVerificationRunner` decides which of the three tiers each operator takes and reports what it could not run and why. `LogicalOp.orderSensitive` and `@SampleColumn` are the two things the operators had to say for this to read them: whether row order is part of the contract, and which column a field should be pointed at when the first unused one would be a poor choice. Most of the operator set does not implement the generator yet — it arrives a family at a time — and the runner reports each of those rather than passing over it. The tier assertions for a family land with the change that gives that family its generator. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…rces

Ten sources say how they read outside the engine, so an exported script starts from the same data the workflow did rather than from a variable the reader has to fill in: the CSV family, JSON Lines, Arrow, plain text, and the two that read a file named at runtime. A source is the one place where the script cannot simply repeat what the operator does. The engine resolves a dataset through Texera's storage and hands the operator a URI; a script has no such resolver, so it reads the file from its own directory by the name the URI ended with. That is what makes an exported script portable, and it is also its one precondition: the data has to sit beside the script. Two are reported as unverifiable rather than exported blind. File Scan takes its filenames from an input port at run time, which a source harness has nothing to feed; URL Fetcher reads a live URL, so no two runs are required to agree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-02T18:20:13Z

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

Contributors with relevant context: @roshiiiz, @eugenegujing, @Ma77Ball
You can notify them by mentioning @roshiiiz, @eugenegujing, @Ma77Ball in a comment.

codecov-commenter · 2026-09-02T18:23:33Z

Codecov Report

❌ Patch coverage is 52.42718% with 98 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.45%. Comparing base (75c85aa) to head (14b5571).

Files with missing lines	Patch %	Lines
...rator/source/scan/json/JSONLScanSourceOpDesc.scala	3.70%	26 Missing ⚠️
...erator/source/scan/file/FileScanSourceOpDesc.scala	48.57%	12 Missing and 6 partials ⚠️
...or/source/scan/csvOld/CSVOldScanSourceOpDesc.scala	5.55%	17 Missing ⚠️
.../source/scan/csv/ParallelCSVScanSourceOpDesc.scala	6.66%	14 Missing ⚠️
...operator/source/scan/arrow/ArrowSourceOpDesc.scala	10.00%	9 Missing ⚠️
...rator/source/scan/text/TextInputSourceOpDesc.scala	59.09%	6 Missing and 3 partials ⚠️
...operator/source/scan/csv/CSVScanSourceOpDesc.scala	88.00%	1 Missing and 2 partials ⚠️
...ber/operator/source/fetcher/URLFetcherOpDesc.scala	93.33%	0 Missing and 1 partial ⚠️
...ber/operator/source/scan/file/FileScanOpDesc.scala	97.43%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8341      +/-   ##
============================================
- Coverage     93.62%   93.45%   -0.17%     
- Complexity     4857     4887      +30     
============================================
  Files          1212     1212              
  Lines         50037    50213     +176     
  Branches       6132     6177      +45     
============================================
+ Hits          46847    46927      +80     
- Misses         1676     1759      +83     
- Partials       1514     1527      +13

Flag	Coverage Δ		*Carryforward flag
access-control-service	`80.18% <ø> (ø)`
agent-service	`99.32% <ø> (ø)`		Carriedforward from 0e549fa
amber	`89.42% <52.42%> (-0.44%)`	⬇️
computing-unit-managing-service	`77.14% <ø> (ø)`
config-service	`87.12% <ø> (ø)`
file-service	`83.65% <ø> (ø)`		Carriedforward from 0e549fa
frontend	`96.16% <ø> (+<0.01%)`	⬆️	Carriedforward from 0e549fa
notebook-migration-service	`83.73% <ø> (ø)`
pyamber	`98.47% <ø> (ø)`		Carriedforward from 0e549fa
workflow-compiling-service	`74.09% <ø> (ø)`

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-09-02T18:27:07Z

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 4 worse · ⚪ 11 noise (<±5%) · 0 without baseline

Compared against main 75c85aa benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
⚪	bs=10 sw=10 sl=64	555	0.339	17,166/24,679/24,679 us	⚪ within ±5% / 🔴 +53.0%
🔴	bs=100 sw=10 sl=64	1,130	0.689	84,040/137,782/137,782 us	🔴 +37.7% / 🔴 +26.7%
🔴	bs=1000 sw=10 sl=64	1,345	0.821	739,539/881,226/881,226 us	🔴 +14.1% / 🟢 +33.0%

Baseline details

Latest main 75c85aa from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	555 tuples/sec	569 tuples/sec	755.28 tuples/sec	-2.5%	-26.5%
bs=10 sw=10 sl=64	MB/s	0.339 MB/s	0.347 MB/s	0.461 MB/s	-2.3%	-26.5%
bs=10 sw=10 sl=64	p50	17,166 us	16,967 us	12,957 us	+1.2%	+32.5%
bs=10 sw=10 sl=64	p95	24,679 us	23,983 us	16,134 us	+2.9%	+53.0%
bs=10 sw=10 sl=64	p99	24,679 us	23,983 us	20,333 us	+2.9%	+21.4%
bs=100 sw=10 sl=64	throughput	1,130 tuples/sec	1,185 tuples/sec	980.1 tuples/sec	-4.6%	+15.3%
bs=100 sw=10 sl=64	MB/s	0.689 MB/s	0.723 MB/s	0.598 MB/s	-4.7%	+15.2%
bs=100 sw=10 sl=64	p50	84,040 us	82,138 us	101,894 us	+2.3%	-17.5%
bs=100 sw=10 sl=64	p95	137,782 us	100,030 us	108,718 us	+37.7%	+26.7%
bs=100 sw=10 sl=64	p99	137,782 us	100,030 us	122,482 us	+37.7%	+12.5%
bs=1000 sw=10 sl=64	throughput	1,345 tuples/sec	1,363 tuples/sec	1,011 tuples/sec	-1.3%	+33.0%
bs=1000 sw=10 sl=64	MB/s	0.821 MB/s	0.832 MB/s	0.617 MB/s	-1.3%	+33.0%
bs=1000 sw=10 sl=64	p50	739,539 us	733,972 us	996,422 us	+0.8%	-25.8%
bs=1000 sw=10 sl=64	p95	881,226 us	772,252 us	1,037,670 us	+14.1%	-15.1%
bs=1000 sw=10 sl=64	p99	881,226 us	772,252 us	1,072,152 us	+14.1%	-17.8%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,360.28,200,128000,555,0.339,17166.36,24679.08,24679.08
1,100,10,64,20,1770.65,2000,1280000,1130,0.689,84040.30,137781.78,137781.78
2,1000,10,64,20,14874.17,20000,12800000,1345,0.821,739538.53,881225.78,881225.78

…f the one without The split this change relies on was declared but never wired. The specs carry `@IntegrationTest` and `build.sbt` reads `WCS_TEST_FILTER` to act on it, but nothing set that variable, so the filter was a no-op and the specs that fork Python ran in the job that provisions none — failing on `No module named 'pandas'` rather than on anything they were testing. The platform job now sets `skip-integration`, which excludes them. The platform-integration job sets `integration-only` and provisions what they need: Python 3.12, amber's requirements, protoc, and the generated proto bindings, which are gitignored and so have to be regenerated before a forked driver can import pyamber. Every step is guarded on the service, so the other entries in that matrix are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…into feat/standalone-sources

… have one The example was an operator another batch gives a generator to, so the assertion held only until that batch landed. A Python UDF holds whatever order these land in: its body is written by whoever drops the operator, so there is nothing for a generator to emit. The word cloud assertion goes for the same reason. It says the operator is withheld, which a later batch stops being true once its placement is seeded, and the prediction op alone already covers what the test is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Everything here that is not a source operator belongs to apache#8327 and was carried only so this branch could compile and run its own tests before that one landed. Reviewing it twice costs more than the red build does: what is left is the thirteen files this change is actually about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This branch held an older copy of both and changes neither. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

carloea2

The source operator exports look good.

carloea2

Reviewed alongside the related export and verification PRs. These findings are based on code inspection and focused Python checks, not a full Scala suite run.

The engine drops and takes before it parses, so a line outside the configured window is never converted and an unparseable one there costs nothing. The export converted first, which raised on a line the run would never have looked at. TextInputSourceOpDesc had the same ordering. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

carloea2

Follow-up review of the series. Runtime findings were checked locally against the current code; deployment routing was checked from configuration.

… does Two divergences in one reader. The parser sets no null value, so only an empty field is null, while pandas reads a list of words as missing by default and turned the country code NA into one. And a blank header position is named by both, differently: the schema calls it column-N, pandas calls it "Unnamed: N", so a downstream operator asked for a column the frame did not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

FileScanOpExec takes the first String field of the tuple, not the first column. A row carrying an id ahead of the path reads the file on the platform, while the export opened the id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…does IOUtils.toString substitutes U+FFFD for a malformed byte, so the executor hands back a string for any response. A strict decode raised instead, failing the whole script on a body the executor reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

carloea2

Further review with local regression tests and connected execution checks.

The source drops and takes on raw lines, before any of them is read as JSON, so a line outside the window costs nothing however malformed it is. The export read the whole file and sliced the frame afterwards, which ended it on a line the workflow skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Matching the placeholder against the index still renames a header the user really did spell `Unnamed: 1`, when it sits at position 1: there the two cases are the same string in the same place. The schema knows which is which, because a blank header is the only one it replaces, and its names are what every downstream operator was configured against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The reader takes the schema's names by position; this still asserted the placeholder-matching form it emitted before, which no longer appears in the generated code at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kz930 · 2026-09-11T01:18:43Z

@carloea2 the findings here are fixed: the CSV missing values and header naming, the file-name field, the JSONL and file-scan windows, and the URL decode. Would you take another look when you have a moment?

kz930 and others added 5 commits September 1, 2026 16:23

Merge remote-tracking branch 'upstream/main' into feat/standalone-sou…

97e05ec

…rces

github-actions Bot assigned kz930 Sep 2, 2026

github-actions Bot added feature dependencies Pull requests that update a dependency file common platform Non-amber Scala service paths labels Sep 2, 2026

kz930 and others added 2 commits September 2, 2026 11:32

Merge remote-tracking branch 'myfork/feat/standalone-verify-harness' …

9be0dbd

…into feat/standalone-sources

github-actions Bot added the ci changes related to CI label Sep 2, 2026

kz930 and others added 3 commits September 2, 2026 16:10

Merge upstream/main

3f07020

chore: leave the two verify files to apache#8327 as well

df10bd8

This branch held an older copy of both and changes neither. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions Bot removed dependencies Pull requests that update a dependency file ci changes related to CI platform Non-amber Scala service paths labels Sep 2, 2026

kz930 mentioned this pull request Sep 2, 2026

A workflow cannot be taken out of Texera as a script that runs on its own #8325

Open

27 tasks

carloea2 approved these changes Sep 4, 2026

View reviewed changes

kz930 marked this pull request as draft September 4, 2026 17:12

carloea2 reviewed Sep 9, 2026

View reviewed changes

Comment thread .../src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanSourceOpDesc.scala Outdated

carloea2 reviewed Sep 9, 2026

View reviewed changes

kz930 and others added 3 commits September 9, 2026 17:49

carloea2 reviewed Sep 10, 2026

View reviewed changes

Comment thread ...src/main/scala/org/apache/texera/amber/operator/source/scan/json/JSONLScanSourceOpDesc.scala

Comment thread ...or/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala Outdated

kz930 and others added 3 commits September 10, 2026 13:03

Merge remote-tracking branch 'upstream/main' into HEAD

14b5571

Navigation Menu

Uh oh!

feat(workflow-operator): export the source operators as Python - #8341

feat(workflow-operator): export the source operators as Python#8341
kz930 wants to merge 19 commits into
apache:mainfrom
kz930:feat/standalone-sources

kz930 commented Sep 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Sep 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

carloea2 left a comment

Uh oh!

carloea2 left a comment

Uh oh!

Uh oh!

carloea2 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

carloea2 left a comment

Uh oh!

Uh oh!

Uh oh!

kz930 commented Sep 11, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

kz930 commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Reviewer Suggestions

Uh oh!

codecov-commenter commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Benchmark changes need a look

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kz930 commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kz930 commented Sep 2, 2026 •

edited

Loading

github-actions Bot commented Sep 2, 2026 •

edited

Loading

codecov-commenter commented Sep 2, 2026 •

edited

Loading

github-actions Bot commented Sep 2, 2026 •

edited

Loading

kz930 commented Sep 11, 2026 •

edited

Loading