Skip to content

ci: stabilize GPU E2E and merge queue#142

Open
rominf wants to merge 5 commits into
mainfrom
ci-shared-windows-strix-runners
Open

ci: stabilize GPU E2E and merge queue#142
rominf wants to merge 5 commits into
mainfrom
ci-shared-windows-strix-runners

Conversation

@rominf

@rominf rominf commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Retry one readiness-stalled real-GPU managed serve after a clean port/VRAM reset while retaining Qwen3.5 coverage; nonzero launch exits still fail immediately.
  • Add first-class flaky-xfail semantics across the harness, platform.json, and consolidated report. Intermittent known bugs may reproduce or XPASS without reddening the lane, while deterministic XPASS remains fatal as stale metadata.
  • Remove stale MI300X EAI-7052 expectations and mark the documented EAI-7333 readiness/inference races flaky.
  • Add a checked-in advanced CodeQL workflow with merge_group support, replacing default setup so required CodeQL checks are produced in the merge queue.
  • Increase the black-box dashboard wait budget from 20 to 30 seconds after the metrics scenario missed a healthy-but-slow frame by the old deadline.

Root cause

PR and merge-group runs exposed three independent CI issues:

  1. Managed Qwen3.5 launches can return successfully with status starting yet intermittently fail to publish the model before the scenario deadline.
  2. The expectation matrix treated documented intermittent EAI-7333 passes as stale XPASS, repeatedly reddening otherwise healthy MI300X runs.
  3. Branch protection required CodeQL contexts that GitHub default setup did not emit for merge_group; the queue timed out twice despite a green repository CI workflow.

The recovered dedicated Strix Halo Windows runner is used unchanged. The temporary organization-runner routing, wrapper-cleanup disablement, and Windows Lemonade EAI-7455 xfails have been removed.

Risk and scope

  • Risk: medium. Changes are limited to CI workflow triggers and non-blocking E2E orchestration/reporting; production CLI behavior is unchanged.
  • CodeQL remains enabled for actions, Python, and Rust through the checked-in advanced workflow.
  • The permanent Windows Lemonade backend-download work remains outside this PR.

Test plan

  • cargo test -p e2e-cucumber --all-targets — 31 passed
  • cargo test -p e2e-report — 34 passed
  • cargo clippy -p e2e-cucumber -p e2e-report --all-targets -- -D warnings
  • Advanced CodeQL passed for actions, Python, and Rust
  • Focused MI300X Qwen3.5 dispatch passed
  • Dedicated Strix Halo Windows lane passes after runner recovery
  • Merge-group run emits and passes all required CodeQL checks

@rominf
rominf force-pushed the ci-shared-windows-strix-runners branch 5 times, most recently from 3ae216c to c2c2274 Compare July 23, 2026 12:30
@rominf rominf changed the title ci: use shared Windows Strix runners ci: stabilize shared GPU E2E runners Jul 23, 2026
@rominf
rominf enabled auto-merge July 23, 2026 14:13
@rominf
rominf added this pull request to the merge queue Jul 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jul 23, 2026
@rominf
rominf added this pull request to the merge queue Jul 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jul 23, 2026
@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

@rominf
rominf enabled auto-merge July 23, 2026 17:40
@rominf
rominf added this pull request to the merge queue Jul 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 23, 2026
@rominf
rominf enabled auto-merge July 23, 2026 18:25
rominf added 4 commits July 24, 2026 12:20
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the ci-shared-windows-strix-runners branch from bb89ce9 to cf10418 Compare July 24, 2026 12:23
@rominf rominf changed the title ci: stabilize shared GPU E2E runners ci: stabilize GPU E2E and merge queue Jul 24, 2026
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf added this pull request to the merge queue Jul 24, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jul 24, 2026
@rominf
rominf added this pull request to the merge queue Jul 24, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jul 24, 2026
@rominf
rominf added this pull request to the merge queue Jul 27, 2026
@rominf
rominf removed this pull request from the merge queue due to a manual request Jul 27, 2026
@rominf
rominf removed this pull request from the merge queue due to a manual request Jul 27, 2026
@rominf
rominf added this pull request to the merge queue Jul 27, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants