feat(source): bound a file scan's window at zero - #8514
Conversation
Every file-scan source takes a Limit and an Offset from ScanSourceOpDesc, and neither said it cannot be negative. Neither means anything below zero, and a negative one is not read the same way twice: the executors take the window with `drop` and `take`, where a negative drop keeps every row and a negative take keeps none, while the scripts the export writes take it with `iloc`, which counts from the end. `offset = -1` is every row on one side and the last row on the other. Declaring the bound on the two fields states what the operators already assume, and the property editor then refuses the value. It reaches CSV, CSVOld, JSONL, Arrow and the file scans at once, all of which inherit the fields. No executor changes: a plan submitted through the API can still carry a negative, and nothing about how one is processed is touched here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
|
@carloea2 a small one, independent of the export series: a file scan's Limit and Offset accepted a negative value, and a negative one is not read the same way twice, |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8514 +/- ##
=========================================
Coverage 93.62% 93.62%
Complexity 4857 4857
=========================================
Files 1212 1212
Lines 50037 50037
Branches 6132 6132
=========================================
+ Hits 46847 46848 +1
+ Misses 1676 1675 -1
Partials 1514 1514
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 402 | 0.245 | 23,376/32,548/32,548 us | 🟢 -9.1% / 🔴 +101.7% |
| 🟢 | bs=100 sw=10 sl=64 | 812 | 0.496 | 120,120/140,297/140,297 us | 🟢 -14.2% / 🔴 +29.0% |
| ⚪ | bs=1000 sw=10 sl=64 | 908 | 0.554 | 1,106,007/1,133,334/1,133,334 us | ⚪ within ±5% / 🔴 +11.0% |
Baseline details
Latest main 75c85aa from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 402 tuples/sec | 426 tuples/sec | 755.28 tuples/sec | -5.6% | -46.8% |
| bs=10 sw=10 sl=64 | MB/s | 0.245 MB/s | 0.26 MB/s | 0.461 MB/s | -5.8% | -46.9% |
| bs=10 sw=10 sl=64 | p50 | 23,376 us | 21,503 us | 12,957 us | +8.7% | +80.4% |
| bs=10 sw=10 sl=64 | p95 | 32,548 us | 35,807 us | 16,134 us | -9.1% | +101.7% |
| bs=10 sw=10 sl=64 | p99 | 32,548 us | 35,807 us | 20,333 us | -9.1% | +60.1% |
| bs=100 sw=10 sl=64 | throughput | 812 tuples/sec | 809 tuples/sec | 980.1 tuples/sec | +0.4% | -17.2% |
| bs=100 sw=10 sl=64 | MB/s | 0.496 MB/s | 0.494 MB/s | 0.598 MB/s | +0.4% | -17.1% |
| bs=100 sw=10 sl=64 | p50 | 120,120 us | 120,537 us | 101,894 us | -0.3% | +17.9% |
| bs=100 sw=10 sl=64 | p95 | 140,297 us | 163,499 us | 108,718 us | -14.2% | +29.0% |
| bs=100 sw=10 sl=64 | p99 | 140,297 us | 163,499 us | 122,482 us | -14.2% | +14.5% |
| bs=1000 sw=10 sl=64 | throughput | 908 tuples/sec | 924 tuples/sec | 1,011 tuples/sec | -1.7% | -10.2% |
| bs=1000 sw=10 sl=64 | MB/s | 0.554 MB/s | 0.564 MB/s | 0.617 MB/s | -1.8% | -10.2% |
| bs=1000 sw=10 sl=64 | p50 | 1,106,007 us | 1,085,513 us | 996,422 us | +1.9% | +11.0% |
| bs=1000 sw=10 sl=64 | p95 | 1,133,334 us | 1,117,886 us | 1,037,670 us | +1.4% | +9.2% |
| bs=1000 sw=10 sl=64 | p99 | 1,133,334 us | 1,117,886 us | 1,072,152 us | +1.4% | +5.7% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,497.29,200,128000,402,0.245,23376.14,32548.45,32548.45
1,100,10,64,20,2461.69,2000,1280000,812,0.496,120120.31,140296.76,140296.76
2,1000,10,64,20,22025.47,20000,12800000,908,0.554,1106006.72,1133334.30,1133334.30
What changes were proposed in this PR?
Every file-scan source takes a Limit and an Offset from
ScanSourceOpDesc, and neither field said it cannot be negative. Both now declareminimum: 0.Neither value means anything below zero, and a negative one is not read the same way twice. The executors take the window with Scala's
dropandtake, where a negative drop keeps every row and a negative take keeps none. The scripts the export writes take the same window withiloc, which counts from the end instead. Sooffset = -1is every row on one side and the last row on the other, andlimit = -1is no rows on one side and all but the last on the other.Declaring the bound states what the operators already assume, and the property editor then refuses the value rather than passing it on. It reaches CSV, CSVOld, JSONL, Arrow and the file scans at once, all of which inherit the two fields.
No executor changes. This is a bound on the form: a plan submitted straight through the API can still carry a negative, and nothing about how one is processed is touched here.
Any related issues, documentation, discussions?
Closes #8513, the task this change is the whole of.
How was this PR tested?
A case in
ScanSourceOpDescSpecvalidates-1,0and5against the generated schema for both fields, through the same validator the property editor uses rather than restating the bound the schema declares. The 163 tests across the scan-source specs pass unchanged.The four numbers quoted above were read off the two runtimes rather than from memory:
Iterator(1,2,3,4,5).drop(-1)keeps all five and.take(-1)keeps none, while pandas'iloc[-1:]is the last row andiloc[:-1]is the first four.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
🤖 Generated with Claude Code