chore(ci): Register the substrait check job with runs-on, route spark checks through xtask - #25195
Open
namanjain24-sudo wants to merge 1 commit into
Open
chore(ci): Register the substrait check job with runs-on, route spark checks through xtask#25195namanjain24-sudo wants to merge 1 commit into
namanjain24-sudo wants to merge 1 commit into
Conversation
namanjain24-sudo
force-pushed
the
ci-align-check-jobs
branch
from
September 11, 2026 16:02
4f28126 to
20ce57d
Compare
… checks through xtask Two small inconsistencies in the `cargo check <crate> features` jobs. `cargo check datafusion-substrait features` selects a `runs-on` runner with `extras=s3-cache`, but is the only one of the fifteen jobs that select such a runner without the `runs-on/action` step that registers it. Without that step the job gets neither the external cache nor the job metrics the other fourteen report. `cargo check datafusion-spark features` is the last place in the workflow that calls `cargo check` directly rather than going through `cargo xtask ci step check`, so those three checks cannot be reproduced locally the way every other check step can. The three commands xtask emits are identical to the ones being replaced: default -> cargo check --profile ci --all-targets -p datafusion-spark no-default -> cargo check --profile ci --no-default-features -p datafusion-spark core -> cargo check --profile ci --no-default-features -p datafusion-spark --features core Job names are unchanged, so the required checks in `.asf.yaml` are unaffected.
namanjain24-sudo
force-pushed
the
ci-align-check-jobs
branch
from
September 11, 2026 17:22
20ce57d to
2b0cfd9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
cargo checkhalf of Add Local Scripts to Reproduce Full CI #21048, which asks for the CI steps to be reproducible locally.Rationale for this change
While measuring the
cargo check <crate> featuresjobs for #25148 I found two inconsistencies in them. Neither is a policy question about which runner to use, so they seemed worth separating out.1. The substrait check job never registers with
runs-on.Fifteen jobs in
rust.ymlselect aruns-onrunner withextras=s3-cache. Fourteen of them start with theruns-on/actionstep that registers the job.cargo check datafusion-substrait featuresis the one that does not, so onpushandmerge_groupit lands on theruns-onrunner but gets neither the external cache nor the job metrics the other fourteen report. It looks like an oversight rather than a choice; the job directly above and below it in the file both have the step.2. The spark check job is the last one not going through
xtask.cargo check datafusion-spark featurescallscargo checkdirectly. Every other check step in the workflow goes throughcargo xtask ci step check, which is what lets a contributor run the same thing locally. After this PR there is no rawcargo checkleft inrust.yml.What changes are included in this PR?
Four lines.
runs-on/actionstep tolinux-datafusion-substrait-features, matching its siblings.datafusion-sparkcargo checkinvocations with thextaskequivalents.The commands
xtaskemits are the same ones being replaced, which--explainconfirms:cargo xtask ci step check datafusion-spark …cargo check --profile ci --all-targets -p datafusion-sparkdefaultcargo check --profile ci --no-default-features -p datafusion-sparkno-defaultcargo check --profile ci --no-default-features -p datafusion-spark --features=corecoreI deliberately left out the larger changes those jobs invite, because they are the open question in #25148 rather than something to decide in a cleanup PR. For the record, what I measured there: five of the seven
cargo check <crate> featuresjobs declare noRust Dependency Cachestep at all, onlydatafusionanddatafusion-substraitdo; three of the seven are pinned toubuntu-latestrather than thevars.USE_RUNS_ONexpression; and across those seven jobs about a third of the wall clock is setup and cache restore paid seven times over. Happy to follow up on any of that if it is wanted.What is the testing strategy for this PR?
The three
xtaskcommands were run locally against this branch and all pass:python3 ci/scripts/check_asf_yaml_status_checks.pyreportsOK: All 32 required_status_checks match existing GitHub Actions jobs— job names are unchanged, so the required checks are unaffected.ci/scripts/check_no_cargo_install_in_workflows.shandci/scripts/typos_check.share clean, and the file still parses as YAML with the same 26 jobs.The
runs-on/actionhalf cannot be verified outside the Apache repo, since the action is a no-op on standard GitHub runners.Are there any user-facing changes?
No. CI configuration only.