Skip to content

ci: parallelize the golden gate and free the package job - #48

Merged
Jo5ta merged 1 commit into
mainfrom
ci/golden-and-test-parallelism
Jul 28, 2026
Merged

ci: parallelize the golden gate and free the package job#48
Jo5ta merged 1 commit into
mainfrom
ci/golden-and-test-parallelism

Conversation

@Jo5ta

@Jo5ta Jo5ta commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

What

Cuts CI wall-clock time by attacking the two things a timing analysis showed dominate it: the golden format gate (~740s) and the serial package tail (~100s).

Why (measured, from a passing run)

Golden was the critical path: two ~6-min qemu-emulated legs (le-aarch64, be-s390x) ran back-to-back in one job — aarch64 367s + s390x 359s = ~726s of pure emulated-build compute — and it ran on every code||tests||ci PR. package then ran serially after build-and-test even though it builds its own rpms.

Changes

  • Split golden by byte order into golden-le + golden-be so the two legs run in parallel (step_golden.sh was already parameterized by CLLTK_GOLDEN_ARCHES). Wall-clock for golden goes from le + be to max(le, be).
  • golden-le runs on a native ARM64 runner (ubuntu-24.04-arm) — no qemu. Only the s390x leg needs emulation (no native s390x runners exist).
  • Path-gate golden on a new golden filter (tracing_library/, the golden generator, the cmake preset, the gate script). The trace format can only change through those, so the slow job is skipped on the majority of PRs.
  • Free package: depend on checks instead of build-and-test. It builds its own rpms (cmake --workflow --preset rpms), so the build-and-test dependency was only a gate; it now runs in parallel. Trade-off: it builds on PRs whose tests fail (cheap; the PR fails anyway).

Expected impact

  • Non-trace PRs: golden skipped entirely → the two ~6-min legs disappear from the run.
  • Trace PRs: golden ≈ the s390x leg alone (~6 min emulated) instead of ~12 min; the LE leg is native and fast.
  • package's ~100s comes off the critical path.

After this, build-and-test (dominated by the ~270s Python suite) becomes the pole — a follow-up will split that into parallel jobs.

⚠️ Branch protection

This renames the golden status check to golden-le / golden-be. If golden is in the required-status-checks list, that list needs updating (replace golden with golden-le and golden-be), or the required check will never report and block merges.

Testing

  • step_golden.sh with CLLTK_GOLDEN_ARCHES=aarch64 (the LE leg) and =s390x (the BE leg) are validated locally; the workflow YAML parses and the job graph is correct. Full behavior confirmed by this PR's own CI run.

The golden format gate was the CI critical path (~740s): two ~6-min
qemu-emulated legs (le-aarch64, be-s390x) ran back to back in one job, on
every code/tests/ci PR.

- Split golden into golden-le + golden-be so the byte orders run in parallel
  (step_golden.sh is already parameterized by CLLTK_GOLDEN_ARCHES). The LE leg
  runs on a native ARM64 runner (ubuntu-24.04-arm), so only the BE (s390x) leg
  needs qemu emulation.
- Gate both on a new 'golden' path filter (tracing_library / golden generator /
  cmake preset / gate script) so the slow job is skipped on the majority of PRs
  that cannot change the trace format.
- package builds its own rpms (cmake --workflow --preset rpms), so depend it on
  'checks' instead of build-and-test; it ran serially after build-and-test
  before (~100s tail on the critical path).

Follow-up: split the ~270s Python test suite (the next critical-path item).
Signed-off-by: Jo5ta <jo5ta@mail.de>
@Jo5ta
Jo5ta force-pushed the ci/golden-and-test-parallelism branch from 5786178 to 80ce7e0 Compare July 28, 2026 14:48
@Jo5ta
Jo5ta merged commit 7726418 into main Jul 28, 2026
25 checks passed
@Jo5ta
Jo5ta deleted the ci/golden-and-test-parallelism branch July 28, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant