Skip to content

feat(source): bound a file scan's window at zero - #8514

Open
kz930 wants to merge 1 commit into
apache:mainfrom
kz930:fix/scan-window-bound
Open

feat(source): bound a file scan's window at zero#8514
kz930 wants to merge 1 commit into
apache:mainfrom
kz930:fix/scan-window-bound

Conversation

@kz930

@kz930 kz930 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Every file-scan source takes a Limit and an Offset from ScanSourceOpDesc, and neither field said it cannot be negative. Both now declare minimum: 0.

Neither value means anything below zero, and a negative one is not read the same way twice. The executors take the window with Scala's drop and take, where a negative drop keeps every row and a negative take keeps none. The scripts the export writes take the same window with iloc, which counts from the end instead. So offset = -1 is every row on one side and the last row on the other, and limit = -1 is no rows on one side and all but the last on the other.

Declaring the bound states what the operators already assume, and the property editor then refuses the value rather than passing it on. It reaches CSV, CSVOld, JSONL, Arrow and the file scans at once, all of which inherit the two fields.

No executor changes. This is a bound on the form: a plan submitted straight through the API can still carry a negative, and nothing about how one is processed is touched here.

Any related issues, documentation, discussions?

Closes #8513, the task this change is the whole of.

How was this PR tested?

A case in ScanSourceOpDescSpec validates -1, 0 and 5 against the generated schema for both fields, through the same validator the property editor uses rather than restating the bound the schema declares. The 163 tests across the scan-source specs pass unchanged.

The four numbers quoted above were read off the two runtimes rather than from memory: Iterator(1,2,3,4,5).drop(-1) keeps all five and .take(-1) keeps none, while pandas' iloc[-1:] is the last row and iloc[:-1] is the first four.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

🤖 Generated with Claude Code

Every file-scan source takes a Limit and an Offset from ScanSourceOpDesc,
and neither said it cannot be negative. Neither means anything below zero,
and a negative one is not read the same way twice: the executors take the
window with `drop` and `take`, where a negative drop keeps every row and a
negative take keeps none, while the scripts the export writes take it with
`iloc`, which counts from the end. `offset = -1` is every row on one side
and the last row on the other.

Declaring the bound on the two fields states what the operators already
assume, and the property editor then refuses the value. It reaches CSV,
CSVOld, JSONL, Arrow and the file scans at once, all of which inherit the
fields. No executor changes: a plan submitted through the API can still
carry a negative, and nothing about how one is processed is touched here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @xuang7, @eugenegujing
    You can notify them by mentioning @xuang7, @eugenegujing in a comment.

@kz930

kz930 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 a small one, independent of the export series: a file scan's Limit and Offset accepted a negative value, and a negative one is not read the same way twice, drop(-1) keeps every row where iloc[-1:] keeps the last. Would you take a look when you have a moment?

@codecov-commenter

codecov-commenter commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.62%. Comparing base (75c85aa) to head (76c98b3).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #8514   +/-   ##
=========================================
  Coverage     93.62%   93.62%           
  Complexity     4857     4857           
=========================================
  Files          1212     1212           
  Lines         50037    50037           
  Branches       6132     6132           
=========================================
+ Hits          46847    46848    +1     
+ Misses         1676     1675    -1     
  Partials       1514     1514           
Flag Coverage Δ *Carryforward flag
access-control-service 80.18% <ø> (ø)
agent-service 99.32% <ø> (ø) Carriedforward from 75c85aa
amber 89.86% <ø> (+<0.01%) ⬆️
computing-unit-managing-service 77.14% <ø> (ø)
config-service 87.12% <ø> (ø)
file-service 83.65% <ø> (ø)
frontend 96.16% <ø> (ø) Carriedforward from 75c85aa
notebook-migration-service 83.73% <ø> (ø)
pyamber 98.47% <ø> (ø) Carriedforward from 75c85aa
workflow-compiling-service 74.09% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 4 better · 🔴 3 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 75c85aa benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 402 0.245 23,376/32,548/32,548 us 🟢 -9.1% / 🔴 +101.7%
🟢 bs=100 sw=10 sl=64 812 0.496 120,120/140,297/140,297 us 🟢 -14.2% / 🔴 +29.0%
bs=1000 sw=10 sl=64 908 0.554 1,106,007/1,133,334/1,133,334 us ⚪ within ±5% / 🔴 +11.0%
Baseline details

Latest main 75c85aa from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 402 tuples/sec 426 tuples/sec 755.28 tuples/sec -5.6% -46.8%
bs=10 sw=10 sl=64 MB/s 0.245 MB/s 0.26 MB/s 0.461 MB/s -5.8% -46.9%
bs=10 sw=10 sl=64 p50 23,376 us 21,503 us 12,957 us +8.7% +80.4%
bs=10 sw=10 sl=64 p95 32,548 us 35,807 us 16,134 us -9.1% +101.7%
bs=10 sw=10 sl=64 p99 32,548 us 35,807 us 20,333 us -9.1% +60.1%
bs=100 sw=10 sl=64 throughput 812 tuples/sec 809 tuples/sec 980.1 tuples/sec +0.4% -17.2%
bs=100 sw=10 sl=64 MB/s 0.496 MB/s 0.494 MB/s 0.598 MB/s +0.4% -17.1%
bs=100 sw=10 sl=64 p50 120,120 us 120,537 us 101,894 us -0.3% +17.9%
bs=100 sw=10 sl=64 p95 140,297 us 163,499 us 108,718 us -14.2% +29.0%
bs=100 sw=10 sl=64 p99 140,297 us 163,499 us 122,482 us -14.2% +14.5%
bs=1000 sw=10 sl=64 throughput 908 tuples/sec 924 tuples/sec 1,011 tuples/sec -1.7% -10.2%
bs=1000 sw=10 sl=64 MB/s 0.554 MB/s 0.564 MB/s 0.617 MB/s -1.8% -10.2%
bs=1000 sw=10 sl=64 p50 1,106,007 us 1,085,513 us 996,422 us +1.9% +11.0%
bs=1000 sw=10 sl=64 p95 1,133,334 us 1,117,886 us 1,037,670 us +1.4% +9.2%
bs=1000 sw=10 sl=64 p99 1,133,334 us 1,117,886 us 1,072,152 us +1.4% +5.7%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,497.29,200,128000,402,0.245,23376.14,32548.45,32548.45
1,100,10,64,20,2461.69,2000,1280000,812,0.496,120120.31,140296.76,140296.76
2,1000,10,64,20,22025.47,20000,12800000,908,0.554,1106006.72,1133334.30,1133334.30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A file scan's Limit and Offset accept a negative value

2 participants