ci: move the Spark 4.1 sql_hive shards behind the merge queue - #5871
Open
andygrove wants to merge 1 commit into
Open
ci: move the Spark 4.1 sql_hive shards behind the merge queue#5871andygrove wants to merge 1 commit into
andygrove wants to merge 1 commit into
Conversation
Keep catalyst and the three sql_core shards in the PR tier and run the three sql_hive shards only in the merge queue, or on a pull request that carries the new run-spark-4.1-hive-tests label. Over Aug 12 to Sep 11 the hive shards cost about 65 runner-hours a day on pull requests and were the only failing job on 7 PR runs, against 33 for the sql_core shards, and their 67-minute shard set the PR tier's wall clock at about 110 minutes. The reusable workflow gains a `modules` input (all, core or hive). The matrix rows move from a workflow literal to dev/ci/spark-sql-modules.py, which the build job runs to publish the selected rows as a job output, because a job-level `if:` cannot see the matrix context. ci.yml folds two POLICY outputs, spark_4_1 and the new queue-only spark_4_1_hive, into that one input so the queue still gets every module from a single build. Part of apache#5870.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #5870 (step 2 of the plan there). It does not close the issue.
Rationale for this change
#5843 left the whole
Spark SQL Tests (Spark 4.1)workflow in the PR tier. Over Aug 12 to Sep 11 (1,288 executed CI runs on pull requests, 168 push-to-main runs as the flake baseline):catalyst+sql_core-1/2/3(+ build)sql_hive-1/2/3The hive shards are the worst value in the PR tier by runner-hours per unique catch, and
sql_hive-2at 67 minutes after the 41 minute build is what sets the PR tier's wall clock. Moving them to the queue saves about 65 runner-hours a day and brings PR feedback down to about 100 minutes, while the 33 catches a month fromsql_corestay on pull requests.What changes are included in this PR?
spark_sql_test_reusable.ymlgains amodulesinput (all,coreorhive, defaultall). The matrix rows move from a workflow literal todev/ci/spark-sql-modules.py; thebuildjob runs it and publishes the selected rows as a job output thatspark-sql-testreads withfromJSON, the same pattern the Iceberg reusable workflow uses for its shards. A job-levelif:cannot readmatrix, which is why the rows are left out rather than skipped.dev/ci/compute-changes.pyadds aspark_4_1_hiveoutput with the same path filters asspark_4_1and POLICY["queue", "label:run-spark-4.1-hive-tests"].spark_4_1stays["pr", "queue"].ci.ymlfolds the two outputs into the onespark_4_1call, so the queue still gets every module from a single build: a plain PR run getscore, a merge group run (or a PR carrying the label) getsall, and alabeledrun for the new label getshiveonly.dev/ci/check-ci-config.pygains the two policy cases for the label and a check thatcoreandhivepartitionall..github/workflows/README.mdanddocs/source/contributor-guide/ci.mddocument the new tier split and label.The
run-spark-4.1-hive-testslabel has been created in the repository.Other Spark versions are unaffected:
spark_3_4,spark_3_5andspark_4_0do not passmodulesand keep running all seven rows in the queue.How are these changes tested?
python3 dev/ci/check-ci-config.pypasses, including the new label cases and the module partition check.actionlintpasses. It is what caught the first attempt, a job-levelif:onmatrix.module.name, which is not a valid context there.dev/ci/spark-sql-modules.py --modules core|hive|allprints the expected rows and rejects anything else.compute()plus theci.ymlexpression: PRsynchronizeselectscore, PR with the label selectsall,labeledwith the new label selectshive,merge_groupselectsall,pushdoes not call the workflow.Spark SQL Tests (Spark 4.1).