Skip to content

ci: move the Spark 4.1 sql_hive shards behind the merge queue #5871

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andygrove merged 1 commit into from
Sep 13, 2026
Merged

ci: move the Spark 4.1 sql_hive shards behind the merge queue #5871

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 24 additions & 23 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ Merging goes through GitHub's merge queue, configured by the `Merge Queue`
ruleset in `.asf.yaml`. That splits CI into two tiers:

- **PR tier** (`pr`): fast feedback while a change is being iterated on.
The Linux build, Spark 4.1 and Iceberg 1.11.
The Linux build, Spark 4.1 (catalyst and `sql_core` only) and Iceberg 1.11.
- **Queue tier** (`queue`): the authoritative gate. Everything the PR tier
runs, plus the macOS build, the benchmark compile check, Spark 3.4/3.5/4.0
and Iceberg 1.8/1.9/1.10, evaluated against the merge result rather than
against the PR head.
runs, plus the macOS build, the benchmark compile check, the Spark 4.1
`sql_hive` shards, Spark 3.4/3.5/4.0 and Iceberg 1.8/1.9/1.10, evaluated
against the merge result rather than against the PR head.

Every queue-only job has a `run-*` label that opts a pull request into it
early, listed in the diagram below.
Expand Down Expand Up @@ -59,8 +59,9 @@ and TPC-H/TPC-DS caches on `main` stale until the next unrelated change.
PR + queue tier push to main only queue tier, or PR with label
--------------- ----------------- ---------------------------
pr_build_linux (+ push, for cache) docs pr_build_macos run-macos-tests
spark_4_1 pr_benchmark_check run-benchmark-check
iceberg_1_11 spark_3_4 run-spark-3.4-tests
spark_4_1 (catalyst + sql_core) pr_benchmark_check run-benchmark-check
iceberg_1_11 spark_4_1 sql_hive run-spark-4.1-hive-tests
spark_3_4 run-spark-3.4-tests
spark_3_5 run-spark-3.5-tests
spark_4_0 run-spark-4.0-tests
iceberg_1_8 run-iceberg-tests
Expand All @@ -85,23 +86,23 @@ and TPC-H/TPC-DS caches on `main` stale until the next unrelated change.

## What runs when

| Job in `ci.yml` | Triggered by | Routing rule |
| -------------------- | ------------------------------------------------- | ----------------------------------- |
| `preflight` | every PR / merge group / push / dispatch / label | none (always runs) |
| `changes` | every PR / merge group / push / dispatch / label | runs `dev/ci/compute-changes.py` |
| `pr_build_linux` | PR, merge group or push to main, paths matched | `dev/ci/compute-changes.py` |
| `pr_build_macos` | merge group, **or** PR with `run-macos-tests` | `dev/ci/compute-changes.py` |
| `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only |
| `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` |
| `spark_3_5` | merge group, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources |
| `spark_4_1` | PR or merge group, paths matched | Spark 4.1 sources |
| `spark_3_4` | merge group, **or** PR with `run-spark-3.4-tests` | Spark 3.4 sources |
| `spark_4_0` | merge group, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources |
| `iceberg_1_11` | PR or merge group, paths matched | Iceberg sources |
| `iceberg_1_8` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_9` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_10` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `required_checks` | always, after every job above except `docs` | none (always runs) |
| Job in `ci.yml` | Triggered by | Routing rule |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| `preflight` | every PR / merge group / push / dispatch / label | none (always runs) |
| `changes` | every PR / merge group / push / dispatch / label | runs `dev/ci/compute-changes.py` |
| `pr_build_linux` | PR, merge group or push to main, paths matched | `dev/ci/compute-changes.py` |
| `pr_build_macos` | merge group, **or** PR with `run-macos-tests` | `dev/ci/compute-changes.py` |
| `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only |
| `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` |
| `spark_3_5` | merge group, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources |
| `spark_4_1` | PR or merge group, paths matched; the `sql_hive` shards only in the merge group **or** with `run-spark-4.1-hive-tests` | Spark 4.1 sources |
| `spark_3_4` | merge group, **or** PR with `run-spark-3.4-tests` | Spark 3.4 sources |
| `spark_4_0` | merge group, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources |
| `iceberg_1_11` | PR or merge group, paths matched | Iceberg sources |
| `iceberg_1_8` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_9` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_10` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `required_checks` | always, after every job above except `docs` | none (always runs) |

A heavy job appears in the PR's checks list as a `skipped` entry whenever
its path filter or event criteria don't match. Skipped checks count as
Expand Down
21 changes: 16 additions & 5 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,12 @@
# Merging goes through GitHub's merge queue (see `rulesets` in `.asf.yaml`), so
# there are two tiers:
#
# pull_request fast feedback. Linux build + Spark 4.1 + Iceberg 1.11.
# pull_request fast feedback. Linux build + Spark 4.1 (catalyst and
# sql_core only) + Iceberg 1.11.
# merge_group the authoritative gate. Everything the PR tier runs, plus
# the macOS build, the benchmark compile check, Spark
# 3.4/3.5/4.0 and Iceberg 1.8/1.9/1.10, evaluated against
# the merge result rather than the PR head.
# the macOS build, the benchmark compile check, the Spark 4.1
# sql_hive shards, Spark 3.4/3.5/4.0 and Iceberg 1.8/1.9/1.10,
# evaluated against the merge result rather than the PR head.
#
# Which tier a job sits in is POLICY in dev/ci/compute-changes.py, not an
# expression here. Heavy jobs deliberately have no `push` tier: the queue
Expand Down Expand Up @@ -160,6 +161,7 @@ jobs:
spark_3_5: ${{ steps.compute.outputs.spark_3_5 }}
spark_4_0: ${{ steps.compute.outputs.spark_4_0 }}
spark_4_1: ${{ steps.compute.outputs.spark_4_1 }}
spark_4_1_hive: ${{ steps.compute.outputs.spark_4_1_hive }}
iceberg_1_8: ${{ steps.compute.outputs.iceberg_1_8 }}
iceberg_1_9: ${{ steps.compute.outputs.iceberg_1_9 }}
iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }}
Expand Down Expand Up @@ -288,12 +290,21 @@ jobs:
spark_4_1:
name: Spark SQL Tests (Spark 4.1)
needs: changes
if: needs.changes.outputs.spark_4_1 == 'true'
# Two POLICY outputs feed one call, so the queue gets every module from a
# single 40-minute build instead of two. `spark_4_1` (PR tier) covers
# catalyst and the sql_core shards; `spark_4_1_hive` (queue-only, or the
# `run-spark-4.1-hive-tests` label) adds the sql_hive shards. A `labeled`
# run sets only the second, and then runs only the hive rows.
if: needs.changes.outputs.spark_4_1 == 'true' || needs.changes.outputs.spark_4_1_hive == 'true'
uses: ./.github/workflows/spark_sql_test_reusable.yml
with:
spark-short: '4.1'
spark-full: '4.1.3'
java: 17
modules: >-
${{ needs.changes.outputs.spark_4_1 != 'true' && 'hive'
|| needs.changes.outputs.spark_4_1_hive != 'true' && 'core'
|| 'all' }}

iceberg_1_8:
name: Iceberg Spark SQL Tests (Iceberg 1.8)
Expand Down
32 changes: 22 additions & 10 deletions .github/workflows/spark_sql_test_reusable.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,17 @@ on:
required: false
default: false
type: boolean
modules:
# Which matrix rows to run. `core` is catalyst plus the three sql_core
# shards, `hive` is the three sql_hive shards, `all` is both. The
# umbrella uses this to keep the Spark 4.1 hive shards out of the PR
# tier without a second call (and a second 40-minute build) for the
# queue: one call, and the rows the tier does not want are left out
# of the matrix by dev/ci/spark-sql-modules.py.
description: 'Which test modules to run: all, core (catalyst + sql_core) or hive (sql_hive)'
required: false
default: 'all'
type: string

env:
RUST_VERSION: stable
Expand Down Expand Up @@ -71,9 +82,15 @@ jobs:
runs-on: ubuntu-24.04
container:
image: amd64/rust
outputs:
module-matrix: ${{ steps.modules.outputs.matrix }}
steps:
- uses: actions/checkout@v7

- name: Select test modules
id: modules
run: python3 dev/ci/spark-sql-modules.py --modules "${{ inputs.modules }}" --github-output "$GITHUB_OUTPUT"

- name: Setup Rust & Java toolchain
uses: ./.github/actions/setup-builder
with:
Expand Down Expand Up @@ -173,16 +190,11 @@ jobs:
spark-sql-test:
needs: build
strategy:
matrix:
module:
- {name: "catalyst", args1: "catalyst/test", args2: ""}
# sql_core-* set HEAP_SIZE / METASPACE_SIZE so SparkBuild.scala caps
- {name: "sql_core-1", args1: "", args2: "sql/testOnly * -- -l org.apache.spark.tags.ExtendedSQLTest -l org.apache.spark.tags.SlowSQLTest", heap: "3g", metaspace: "1g"}
- {name: "sql_core-2", args1: "", args2: "sql/testOnly * -- -n org.apache.spark.tags.ExtendedSQLTest", heap: "3g", metaspace: "1g"}
- {name: "sql_core-3", args1: "", args2: "sql/testOnly * -- -n org.apache.spark.tags.SlowSQLTest", heap: "3g", metaspace: "1g"}
- {name: "sql_hive-1", args1: "", args2: "hive/testOnly * -- -l org.apache.spark.tags.ExtendedHiveTest -l org.apache.spark.tags.SlowHiveTest"}
- {name: "sql_hive-2", args1: "", args2: "hive/testOnly * -- -n org.apache.spark.tags.ExtendedHiveTest"}
- {name: "sql_hive-3", args1: "", args2: "hive/testOnly * -- -n org.apache.spark.tags.SlowHiveTest"}
# The rows are defined in dev/ci/spark-sql-modules.py and selected by
# the `modules` input; the build job runs the script and publishes the
# result. A job-level `if:` cannot read `matrix`, so a row the caller
# does not want has to be left out of the matrix rather than skipped.
matrix: ${{ fromJSON(needs.build.outputs.module-matrix) }}
fail-fast: false
name: spark-sql-${{ matrix.module.name }}/spark-${{ inputs.spark-full }}-jdk${{ inputs.java }}
runs-on: ubuntu-24.04
Expand Down
47 changes: 46 additions & 1 deletion dev/ci/check-ci-config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@
"spark_3_5",
"spark_4_0",
"spark_4_1",
"spark_4_1_hive",
"iceberg_1_8",
"iceberg_1_9",
"iceberg_1_10",
Expand Down Expand Up @@ -113,7 +114,7 @@
# derived from POLICY, so that a change to the routing has to be stated twice
# and cannot be made by accident.
PR_TIER = {"build_linux", "spark_4_1", "iceberg_1_11"}
SPARK_OPT_IN = {"spark_3_4", "spark_3_5", "spark_4_0"}
SPARK_OPT_IN = {"spark_3_4", "spark_3_5", "spark_4_0", "spark_4_1_hive"}
ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10"}
BUILD_OPT_IN = {"build_macos", "benchmark"}
QUEUE_TIER = PR_TIER | SPARK_OPT_IN | ICEBERG_OPT_IN | BUILD_OPT_IN
Expand Down Expand Up @@ -157,6 +158,21 @@
},
{"build_macos"},
),
# The Spark 4.1 hive shards are queue-only with their own label. The label
# adds them to the PR tier's Spark 4.1 call rather than starting a second.
(
{"name": "pull_request", "action": "synchronize", "labels": ["run-spark-4.1-hive-tests"]},
PR_TIER | {"spark_4_1_hive"},
),
(
{
"name": "pull_request",
"action": "labeled",
"label": "run-spark-4.1-hive-tests",
"labels": ["run-spark-4.1-hive-tests"],
},
{"spark_4_1_hive"},
),
# An opt-in label present on a pushed commit adds just that suite.
(
{"name": "pull_request", "action": "synchronize", "labels": ["run-spark-3.4-tests"]},
Expand Down Expand Up @@ -229,6 +245,34 @@ def load_filters():
return module


def check_spark_sql_modules():
"""`--modules core` and `--modules hive` must partition `--modules all`.

ci.yml maps its two Spark 4.1 POLICY outputs onto these three values, so a
row that lands in no group, or in both, would either never run or run
twice in the queue, and nothing else would notice.
"""
spec = importlib.util.spec_from_file_location("spark_sql_modules", "dev/ci/spark-sql-modules.py")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
failures = []
names = lambda rows: [row["name"] for row in rows]
everything = names(module.select("all"))
core, hive = names(module.select("core")), names(module.select("hive"))
if not core or not hive:
failures.append("a module group is empty (see MODULES in dev/ci/spark-sql-modules.py)")
if sorted(core + hive) != sorted(everything):
failures.append(
f"core {core} + hive {hive} does not partition all {everything} "
f"(see MODULES in dev/ci/spark-sql-modules.py)"
)
if len(set(everything)) != len(everything):
failures.append(f"duplicate module names in {everything}")
for failure in failures:
print(f"spark sql modules: {failure}")
return not failures


def check_change_filters():
module = load_filters()
failures = []
Expand Down Expand Up @@ -545,6 +589,7 @@ def check_required_checks():
if __name__ == "__main__":
ok = check_change_filters()
ok = check_event_policy() and ok
ok = check_spark_sql_modules() and ok
ok = check_artifact_names() and ok
ok = check_local_actions_have_checkout() and ok
ok = check_required_checks() and ok
Expand Down
17 changes: 16 additions & 1 deletion dev/ci/compute-changes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@
"rust-toolchain.toml",
".github/workflows/ci.yml",
".github/workflows/spark_sql_test_reusable.yml",
"dev/ci/spark-sql-modules.py",
".github/actions/setup-builder/**",
".github/actions/setup-spark-builder/**",
".github/actions/upload-artifact-retry/**",
Expand All @@ -151,6 +152,7 @@
"rust-toolchain.toml",
".github/workflows/ci.yml",
".github/workflows/spark_sql_test_reusable.yml",
"dev/ci/spark-sql-modules.py",
".github/actions/setup-builder/**",
".github/actions/setup-spark-builder/**",
".github/actions/upload-artifact-retry/**",
Expand All @@ -177,6 +179,7 @@
"rust-toolchain.toml",
".github/workflows/ci.yml",
".github/workflows/spark_sql_test_reusable.yml",
"dev/ci/spark-sql-modules.py",
".github/actions/setup-builder/**",
".github/actions/setup-spark-builder/**",
".github/actions/upload-artifact-retry/**",
Expand All @@ -203,13 +206,19 @@
"rust-toolchain.toml",
".github/workflows/ci.yml",
".github/workflows/spark_sql_test_reusable.yml",
"dev/ci/spark-sql-modules.py",
".github/actions/setup-builder/**",
".github/actions/setup-spark-builder/**",
".github/actions/upload-artifact-retry/**",
".github/actions/download-artifact-retry/**",
".mvn/**",
"mvnw",
],
# Same inputs as spark_4_1: this is not a separate job but a second
# POLICY decision for the same call, selecting the sql_hive matrix rows.
# ci.yml folds the two outputs into the reusable workflow's `modules`
# input. Populated below, after the dict, so the two lists cannot drift.
"spark_4_1_hive": [],
"iceberg_1_8": [
"native/**/src/**",
"native/**/Cargo.toml",
Expand Down Expand Up @@ -307,6 +316,7 @@
"mvnw",
],
}
FILTERS["spark_4_1_hive"] = FILTERS["spark_4_1"]

# Which events may run each job, independent of the path filters above.
#
Expand Down Expand Up @@ -349,8 +359,13 @@
"spark_3_5": ["queue", "label:run-spark-3.5-tests"],
"spark_4_0": ["queue", "label:run-spark-4.0-tests"],
# Spark 4.1 is the default build profile, so it is the cheapest early
# warning that a change is wrong and stays in the PR tier.
# warning that a change is wrong and stays in the PR tier. Only the
# catalyst and sql_core shards, though: over Aug 12 to Sep 11 2026 the
# three sql_hive shards cost about 65 runner-hours a day on pull requests
# and were the only failing job on 7 PR runs, against 33 for sql_core, and
# their 67-minute shard set the PR tier's wall clock. See issue #5870.
"spark_4_1": ["pr", "queue"],
"spark_4_1_hive": ["queue", "label:run-spark-4.1-hive-tests"],
"iceberg_1_8": ["queue", "label:run-iceberg-tests"],
"iceberg_1_9": ["queue", "label:run-iceberg-tests"],
"iceberg_1_10": ["queue", "label:run-iceberg-tests"],
Expand Down
Loading
Loading