Skip to content

ci: move the Spark 4.1 sql_hive shards behind the merge queue - #5871

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:ci/spark-sql-hive-queue-tier
Open

ci: move the Spark 4.1 sql_hive shards behind the merge queue#5871
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:ci/spark-sql-hive-queue-tier

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #5870 (step 2 of the plan there). It does not close the issue.

Rationale for this change

#5843 left the whole Spark SQL Tests (Spark 4.1) workflow in the PR tier. Over Aug 12 to Sep 11 (1,288 executed CI runs on pull requests, 168 push-to-main runs as the flake baseline):

Spark 4.1 modules Runner hrs/day on PRs PR fail Main fail Runs where it was the only failing job Wall clock
catalyst + sql_core-1/2/3 (+ build) ~96 8.5% 4.5% 33 100 min
sql_hive-1/2/3 ~65 1.4% 1.3% 7 110 min

The hive shards are the worst value in the PR tier by runner-hours per unique catch, and sql_hive-2 at 67 minutes after the 41 minute build is what sets the PR tier's wall clock. Moving them to the queue saves about 65 runner-hours a day and brings PR feedback down to about 100 minutes, while the 33 catches a month from sql_core stay on pull requests.

What changes are included in this PR?

  • spark_sql_test_reusable.yml gains a modules input (all, core or hive, default all). The matrix rows move from a workflow literal to dev/ci/spark-sql-modules.py; the build job runs it and publishes the selected rows as a job output that spark-sql-test reads with fromJSON, the same pattern the Iceberg reusable workflow uses for its shards. A job-level if: cannot read matrix, which is why the rows are left out rather than skipped.
  • dev/ci/compute-changes.py adds a spark_4_1_hive output with the same path filters as spark_4_1 and POLICY ["queue", "label:run-spark-4.1-hive-tests"]. spark_4_1 stays ["pr", "queue"].
  • ci.yml folds the two outputs into the one spark_4_1 call, so the queue still gets every module from a single build: a plain PR run gets core, a merge group run (or a PR carrying the label) gets all, and a labeled run for the new label gets hive only.
  • dev/ci/check-ci-config.py gains the two policy cases for the label and a check that core and hive partition all.
  • .github/workflows/README.md and docs/source/contributor-guide/ci.md document the new tier split and label.

The run-spark-4.1-hive-tests label has been created in the repository.

Other Spark versions are unaffected: spark_3_4, spark_3_5 and spark_4_0 do not pass modules and keep running all seven rows in the queue.

How are these changes tested?

  • python3 dev/ci/check-ci-config.py passes, including the new label cases and the module partition check.
  • actionlint passes. It is what caught the first attempt, a job-level if: on matrix.module.name, which is not a valid context there.
  • dev/ci/spark-sql-modules.py --modules core|hive|all prints the expected rows and rejects anything else.
  • Simulated the routing for each event through compute() plus the ci.yml expression: PR synchronize selects core, PR with the label selects all, labeled with the new label selects hive, merge_group selects all, push does not call the workflow.
  • This PR's own CI run should show the four core shards and no hive shards under Spark SQL Tests (Spark 4.1).

Keep catalyst and the three sql_core shards in the PR tier and run the
three sql_hive shards only in the merge queue, or on a pull request that
carries the new run-spark-4.1-hive-tests label.

Over Aug 12 to Sep 11 the hive shards cost about 65 runner-hours a day on
pull requests and were the only failing job on 7 PR runs, against 33 for
the sql_core shards, and their 67-minute shard set the PR tier's wall
clock at about 110 minutes.

The reusable workflow gains a `modules` input (all, core or hive). The
matrix rows move from a workflow literal to dev/ci/spark-sql-modules.py,
which the build job runs to publish the selected rows as a job output,
because a job-level `if:` cannot see the matrix context. ci.yml folds two
POLICY outputs, spark_4_1 and the new queue-only spark_4_1_hive, into
that one input so the queue still gets every module from a single build.

Part of apache#5870.
@andygrove andygrove added the area:ci CI/CD, GitHub Actions, build tooling label Sep 11, 2026
@github-actions github-actions Bot added build Build environment enhancement New feature or request labels Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ci CI/CD, GitHub Actions, build tooling build Build environment enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant