chore: [branch-1.0] improve network retry configuration for maven and artifact upload (#5782) - #5833
Merged
Conversation
…oad (apache#5782) (cherry picked from commit 80c8ec3) Adapted for branch-1.0, all three in conflict resolution only: - `.github/actions/java-test/action.yaml`: kept branch-1.0's `if: ${{ inputs.upload-test-reports == 'true' }}` on the test-reports upload. The `!cancelled() &&` guard on main comes from a later commit, not from apache#5782, which only bumps that step v6 -> v7. - `.github/workflows/ci.yml`: added only the `check-ci-config.py` preflight step. The three neighbouring steps on main run `check-benchmark-runner.py`, `pr-type-label.test.mjs` and `test-iceberg-shards.py`, none of which exist on branch-1.0. - `dev/ci/compute-changes.py`: added only this commit's three filter entries (`.github/actions/upload-artifact-retry/**`, `.mvn/**`, `mvnw`). The `dev/ci/iceberg-test-shards.gradle`, `check-iceberg-shards.py` and `test-iceberg-shards.py` entries in the same hunks refer to files that do not exist on branch-1.0. The resulting diffstat is identical to the original commit's.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #5782 to
branch-1.0.Cherry-picked from
80c8ec37af422db1952a1128cb23043c9bbfbb72. The resulting diffstat is identicalto the original commit's; the three conflicts were all context drift and are described under
"What changes are included" below.
Prompted by CI on #5820, where
Spark 4.2, JDK 17 [shuffle]failed before running a single test:Maven could not resolve
git-commit-id-maven-plugin's dependencies becausehttps://repo.eclipse.org/...returnedNetwork is unreachable, and the default resolver gives upafter 3 attempts. That is exactly the class of flake this change is meant to absorb, and
branch-1.0does not have it.Which issue does this PR close?
Part of #5489 on
branch-1.0. Listed as a backport candidate in #5815.Rationale for this change
branch-1.0will keep receiving 1.0.x backports for a while, and every one of them pays for thetwo unretried network flakes that #5782 fixed on
main:429/503as retryable, so a502/504from the CDN in front of Maven Central, or a plain unreachable host, fails the buildon first hit.
branch-1.0pins the same Maven 3.9.6 asmain, so it has the same defaults.@actions/artifactclassifies a403fromFinalizeArtifactasnon-retryable and fails the job even though the content uploaded fine.
Re-running a flaked job is a poor substitute here: the Spark SQL and Iceberg jobs are 30-70 minutes
each, and on a release branch a spurious red is easy to mistake for a real backport regression.
What changes are included in this PR?
The change is the original one — see #5782 for the full rationale, including why the Wagon transport
was evaluated and rejected. In short:
.mvn/maven.config(new) raises resolver retries 3 -> 6 andwidens the retryable statuses to
408,429,500,502,503,504;.github/actions/upload-artifact-retry(new) wrapsactions/upload-artifactwith three attempts;artifact names become per-producer;
dev/ci/compute-changes.pylearns the new paths; anddev/ci/check-ci-config.py(new, run frompreflight) pins both invariants.Three conflicts, all resolved by keeping
branch-1.0's surroundings and taking only what #5782itself changed:
.github/actions/java-test/action.yaml: keptbranch-1.0'sif: ${{ inputs.upload-test-reports == 'true' }}on the test-reports upload. The!cancelled() &&guard that appears on
maincomes from a later commit and is only context in chore: Improve network retry configuration for maven and artifact upload #5782's diff, whichbumps that step
v6 -> v7and nothing else..github/workflows/ci.yml: added only thecheck-ci-config.pypreflight step. The three stepsadjacent to it on
mainruncheck-benchmark-runner.py,pr-type-label.test.mjsandtest-iceberg-shards.py, none of which exist onbranch-1.0.dev/ci/compute-changes.py: added only this commit's three filter entries(
.github/actions/upload-artifact-retry/**,.mvn/**,mvnw). Thedev/ci/iceberg-test-shards.gradle,check-iceberg-shards.pyandtest-iceberg-shards.pyentries in the same four hunks refer to files that do not exist on
branch-1.0.Nothing was dropped as inapplicable. The preconditions the change is built around all hold here:
ci.ymlcallsspark_sql_test_reusable.ymlandiceberg_spark_test_reusable.ymlfour times each,which is the shared-artifact-name hazard the per-producer naming addresses;
compute-changes.pyhas exactly the twelve
FILTERSkeyscheck-ci-config.py's routing table expects; and all sixsites that now use the local
upload-artifact-retryaction are in jobs that check out therepository, while
merge-fallback-logs(which deliberately does not) correctly stays on the plainaction.
How are these changes tested?
Verified locally on
branch-1.0with Maven 3.9.6 / JDK 17:help:evaluatereturns6,408,429,500,502,503,504,30000and600000. Also checked fromspark/via../mvnw, so thewrapper's
maven.multiModuleProjectDirectorypinning works here as it does onmain.dev/ci/check-ci-config.pypasses, and is not passing vacuously: itsBUILD_JOBSset matchesbranch-1.0'sFILTERSkeys exactly, so every routing case is actually evaluated.dev/ci/check-suites.py,actionlint -color --shellcheck=offandapache-rat:check(0 unknown licenses) all pass.
git diff --statagainst the original commit is byte-identical, and a diff of the two patchesshows no content delta beyond the three adaptations above.
The retry paths only execute on a real flake, so CI here should show attempt 1 succeeding.
Are there any user-facing changes?
No. CI configuration only; nothing ships in a release artifact.