[release/10.0.1xx] [devops] Provision Xcode from an immutable Azure Artifacts package instead of Provisionator - #26329
Merged
Merged
Conversation
…stead of Provisionator (#26309) CI gets Xcode from Provisionator today. That requires minting two read-only SAS tokens for the `bosstoragemirror` storage account through an Azure service connection, bootstrapping the Provisionator binary from that same account, and trusting whatever bundle comes back — nothing checks that the installed Xcode is the one `Make.config` asks for. This replaces that with an immutable **Azure Artifacts Universal Package** (`xcode-apple-silicon`) pinned in `Make.config`, downloaded with the pipeline's own build identity, and installed by a repository-owned script that verifies it end to end. No Azure subscription service connection, storage account, SAS token, GitHub PAT, or Provisionator bootstrap is involved any more. - **Reproducibility.** The Xcode we build with is now a pinned, immutable coordinate (`xcode-apple-silicon` `26.6.0`, build `17F113`) that is verified against `Make.config` instead of an opaque `.xip` behind a mutable URL. - **Fewer moving parts.** The old path needed a service connection, a storage account with `allowSharedKeyAccess=false`, short-lived user-delegation SAS tokens, `az` on the agent, and the Provisionator channel. All of that is gone. - **Verification.** Nothing previously checked that the provisioned Xcode matched the pinned version. It does now, before anything privileged runs. - **It works in both organizations.** The same change has to run under DevDiv today and dnceng as that migration lands. `templates/common/provision-xcode.yml` replaces `templates/common/provision.yml`: 1. **Resolve** — require Apple silicon, read the pinned coordinates from `Make.config`, and run `install-xcode.sh verify`. If the agent already has that exact Xcode, the download is skipped entirely. 2. **Download** — fetch the Universal Package from the feed belonging to the organization the build is running in. `System.AccessToken` is mapped into *this step only*, and no step in the template enables `set -x`. 3. **Provision** — run `system-dependencies.sh`, which hands the package to `install-xcode.sh` and then provisions Xcode components, simulators and build tools. 4. **Clean** — remove the download, `condition: always()`. `install-xcode.sh` validates the package manifest (schema, size, SHA-256, Apple XIP signature), expands it, and checks the resulting bundle's identifier, marketing version, product build, host architecture, minimum macOS version and Apple code signature **before** anything privileged happens. It installs atomically with backup/rollback, then reconciles `xcode-select`, the `/Applications/Xcode.app` symlink, the Xamarin/MAUI `AppleSdkRoot` settings, the licence and first-launch state. Trust is anchored in **Apple's own signature** and in the **immutability of the feed version** — not in the manifest that ships inside the package (which an attacker who could alter the payload could also alter). The feed name and package name are identical in both organizations; only the organization and the project that hosts the feed differ: | Context | Feed | | --- | --- | | DevDiv | `https://devdiv.visualstudio.com/DevDiv/_artifacts/feed/macios-tools-internal/UPack/xcode-apple-silicon/` | | dnceng | `https://dnceng.visualstudio.com/internal/_artifacts/feed/macios-tools-internal/UPack/xcode-apple-silicon/` | `download-xcode-package.sh` takes the organization from `SYSTEM_COLLECTIONURI` and maps it to the project that owns the feed — `devdiv` → `DevDiv`, `dnceng` → `internal`. Both the legacy `<org>.visualstudio.com` and modern `dev.azure.com/<org>` forms are accepted and normalised to one canonical endpoint per organization, so the result never depends on which form the agent happens to report. Deriving the project from an explicit mapping rather than from `System.TeamProject` keeps the feed address independent of where the pipeline itself runs, and turns an unsupported organization into an actionable error instead of a confusing 404. The mapping is a **default, not a policy**: `--organization`, `--project` and `--feed` all override it, so another organization can be onboarded without touching the script. Note that "dnceng" means the **`internal`** project. `dnceng/public` has no such feed and no access to it. - `templates/common/provision.yml` (SAS minting + Provisionator bootstrap) - `scripts/provisionator-bootstrap.sh` - `build-provisioning.csx`, `provision-xcode.csx`, `provision-brew-packages.csx`, `provision-shared.in.csx`, and the `provisioning` make target that generated `provision-shared.csx` - The `provisionatorChannel` parameter and `PROVISIONATOR_CHANNEL` variable, threaded through every entry pipeline - `ensure-xcode-select.sh` — it hard-failed when `/Applications/Xcode.app` was missing, which blocked the installer that repairs exactly that state - The last use of the `Xamarin - RelEng (BosStorageMirror-Contributor-MI)` service connection from the macOS jobs `XCODE_URL` stays in `Make.config` because local provisioning has no access to the internal feed. **CI no longer reads it**, and refuses to fall back to it under Azure Pipelines so the dependency can't quietly come back. - **Provisioning is expressed once.** Build and simulator tests previously ran Provisionator *and then* `system-dependencies.sh` with a different set of flags. They're now a single call that always passes `--provision-xcode`, so selection, the `Xcode.app` symlink, `AppleSdkRoot` and first-launch state are reconciled. That matters for the Windows reservation job, which deletes those settings immediately beforehand and previously never had them rewritten. - **Brew packages.** `shellcheck`, `yamllint` and `p7zip` move from the Provisionator brew step to `--provision-shellcheck --provision-yamllint --provision-7z`, which run the same `brew install` commands. **`azure-cli` is dropped** — it existed for the SAS generation this PR removes, and the only remaining macOS consumer of `az` (`windows/generate-token.yml`) installs it itself via `azure-tools/az-client-update.yml@yaml-templates`, *before* provisioning runs. It never depended on the brew-installed copy. - **Coverage is a superset.** The reservation job now also gets shellcheck/yamllint/7z, which it previously skipped. It still skips Xcode components, via a new `provisionXcodeComponents` parameter. - **Apple silicon is now required and demanded.** The package contains arm64 slices only. Jobs that provision Xcode gained an explicit `macOS.Architecture -equals arm64` demand so an Intel agent is never scheduled, rather than failing once provisioning starts. The macOS **test** legs are unaffected — they never provision Xcode, so the x64 Sonoma leg keeps running on x64. - **`check_xcode` now runs end to end on build bots for the first time.** `--ignore-all` previously left `IGNORE_XCODE=1`; `--provision-xcode` now clears it. That newly enables `install_coresimulator` and the `iPhoneSimulator`/`MacOSX`/`AppleTVOS` SDK assertions on the build stage. Those SDKs ship inside the bundle, but **a stale `*_NUGET_OS_VERSION` in `Make.versions` will now fail the build stage rather than only the tests.** The old path used a short-lived, read-only, container-scoped SAS from a dedicated managed identity. The new path uses the pipeline's own build identity against a project-scoped feed. That identity was already exposed to PR-controlled steps on the untrusted macOS pool before this PR (`build/build.yml`, `mac/build.yml`, `common/setup.yml` all map `System.AccessToken`), so this doesn't create the exposure — but it does mean the token now also confers feed read access. Secret handling in the new scripts was reviewed specifically: - `SYSTEM_ACCESSTOKEN` is mapped into exactly one step. - No new script or step enables `set -x`. - The token reaches `curl` via `--config -` on **stdin**, and ArtifactTool via `--patvar` (by variable *name*), so it never appears on a command line or in a temp file. - [x] Confirm the `macios-tools-internal` feeds have **no upstream sources**. The Build Service holds the AzDO-default *Feed and Upstream Reader (Collaborator)* role, which cannot publish — but it can save packages from an upstream, and an upstream that also serves `xcode-apple-silicon` would be the one route to shadowing the package. - [ ] Optionally downgrade the Build Service to **Feed Reader** on both feeds; downloading is all these scripts need. - [ ] Consider setting job authorization scope to *current project* for the DevDiv pipelines. - [ ] Keep pinned Xcode coordinates protected from retention deletion — Universal Package versions are immutable and cannot be republished once removed. At the time of writing, both feeds exist and contain `xcode-apple-silicon` `26.6.0`, and neither has retention enabled. The seven commits are ordered so they can be read in sequence, and no file is split across commits: 1. **Add the pinned Xcode Universal Package and its installer** — additive only; `Make.config` coordinates plus `install-xcode.sh`, `packagekit-extract.m`, `download-universal-package.sh`. Nothing calls them yet. 2. **Resolve the Xcode package feed per Azure DevOps organization** — the DevDiv/dnceng mapping, in one file. 3. **Require Apple silicon for the macOS jobs that provision Xcode** — pool demands, four one-line changes. 4. **Install Xcode from a Universal Package** — the `system-dependencies.sh` changes. 5. **Provision Xcode from Azure Artifacts instead of Provisionator** — the actual switch. 6. **Remove the Provisionator and BosStorageMirror plumbing** — pure deletion of what 5 made unreferenced. 7. **Document artifact-based Xcode provisioning** — README and the Xcode bump skill. - `shellcheck` and `yamllint` gates for `tools/devops` pass; `bash -n` clean on every changed script. - **Zero new shellcheck warnings** against a pristine `origin/main` baseline for `system-dependencies.sh` (two pre-existing warnings were removed). - **Zero Azure Pipelines template-contract regressions**, checked by parsing all 64 templates and diffing declared-vs-supplied parameters against a pristine `origin/main` baseline. - Feed resolution verified for all four organization URL forms plus the error paths (unknown organization, missing organization, malformed URL) and the `--project`/`--feed` overrides. - `install-xcode.sh` exercised locally: `verify` success and failure paths, the `Make.config` drift assertion (fires on a mismatch, no false positive for betas), and version/app-name derivation checked against the real `Xcode_26.4.0`, `26.5.0` and `27.0.0-beta4` bundles on disk. - `packagekit-extract.m` compiles and loads against the private `PackageKit` framework. - The Command Line Tools catalog logic unit-tested across all branches (nothing pending, query timeout, exact match, near-miss `26.6.1`, unrelated labels). - **`packagekit-extract.m` uses a private framework.** It declares `PKSignedContainer` by hand, so an OS update could break it at runtime even though it still compiles. `install-xcode.sh` therefore falls back to the supported `xip --expand`, which is not an `xcrun` shim and keeps working. Already-provisioned bots are unaffected either way, since `verify`/`reconcile` never touch PackageKit. - **`softwareupdate` cannot report failure.** It exits `0` for an unknown label, an invalid flag *and* a failed query, and can hang when the update service is unreachable. Command Line Tools installation is therefore best-effort: the catalog query is bounded with a timeout and decided from its contents, and the log says "could not confirm" rather than claiming Apple doesn't offer the label, because a reachable-but-empty catalog is genuinely indistinguishable from a failed query. - The first run on any bot whose `/Applications/Xcode_<version>.app` is a symlink rather than a real directory will reinstall Xcode, because the installer requires a real bundle. - `install-xcode.sh` refuses to replace a real directory at `/Applications/Xcode.app`; it only manages that path as a symlink. Revert the whole range. Commits 5 and 6 are the behavioural switch; 1–4 are inert without them. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e134d13d-5eee-4266-ba75-24422addd02b (cherry picked from commit fcc1dff)
Contributor
There was a problem hiding this comment.
Pull request overview
Backports the devops change to provision Xcode in CI from an immutable Azure Artifacts Universal Package (instead of Provisionator) on release/10.0.1xx, using repo-owned resolver/installer scripts and updated pipeline templates.
Changes:
- Replace Provisionator-based provisioning with
templates/common/provision-xcode.yml+UniversalPackages@0download + verified install viainstall-xcode.sh. - Remove Provisionator scripts/templates/variables and update all pipeline entry points to use the new template.
- Update
system-dependencies.shandMake.configto support--xcode-package-directoryand pinned Xcode package coordinates.
Reviewed changes
Copilot reviewed 34 out of 35 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/devops/README.md | Updates documentation/diagram to reflect new Xcode provisioning flow. |
| tools/devops/provision-xcode.csx | Removes obsolete Provisionator csx entrypoint. |
| tools/devops/provision-shared.in.csx | Removes Provisionator shared provisioning logic. |
| tools/devops/provision-brew-packages.csx | Removes obsolete Provisionator brew provisioning script. |
| tools/devops/Makefile | Removes generation targets for Provisionator csx files. |
| tools/devops/build-provisioning.csx | Removes obsolete Provisionator build provisioning script. |
| tools/devops/automation/templates/windows/reserve-mac.yml | Switches Windows “reserve mac” flow to provision-xcode.yml. |
| tools/devops/automation/templates/variables/common.yml | Removes Provisionator channel variable wiring. |
| tools/devops/automation/templates/tests/build.yml | Replaces Provisionator provisioning with provision-xcode.yml before ./configure. |
| tools/devops/automation/templates/tests-stage.yml | Removes Provisionator channel parameter; adds arm64 agent demand. |
| tools/devops/automation/templates/pipelines/run-tests-pipeline.yml | Removes Provisionator channel parameter plumb-through. |
| tools/devops/automation/templates/pipelines/build-pipeline.yml | Removes Provisionator channel parameter plumb-through. |
| tools/devops/automation/templates/pipelines/api-diff-pipeline.yml | Removes Provisionator channel parameter plumb-through. |
| tools/devops/automation/templates/main-stage.yml | Removes Provisionator channel parameter. |
| tools/devops/automation/templates/mac/build.yml | Removes Provisionator cache/provision steps from Mac build template. |
| tools/devops/automation/templates/common/setup.yml | Drops ensure-xcode-select step; documents selection now handled by Xcode provisioning template. |
| tools/devops/automation/templates/common/provision.yml | Removes old Provisionator+SAS-token provisioning template. |
| tools/devops/automation/templates/common/provision-xcode.yml | Adds new template to resolve/download/install pinned Xcode + deps using Universal Packages. |
| tools/devops/automation/templates/build/build.yml | Replaces Provisionator provisioning with provision-xcode.yml. |
| tools/devops/automation/templates/build/build-stage.yml | Adds arm64 agent demand for Apple-silicon-only Xcode package. |
| tools/devops/automation/templates/build/build-mac-tests-stage.yml | Adds arm64 agent demand for Apple-silicon-only Xcode package. |
| tools/devops/automation/templates/build/api-diff-stage.yml | Adds arm64 agent demand for Apple-silicon-only Xcode package. |
| tools/devops/automation/templates/api-diff-stage.yml | Removes Provisionator channel parameter. |
| tools/devops/automation/scripts/provisionator-bootstrap.sh | Removes Provisionator bootstrap script. |
| tools/devops/automation/scripts/bash/resolve-xcode-package.sh | Adds resolver to map org→project and output feed coordinates for UniversalPackages task. |
| tools/devops/automation/scripts/bash/install-xcode.sh | Adds end-to-end Xcode package/bundle verification + atomic install + reconciliation logic. |
| tools/devops/automation/scripts/bash/ensure-xcode-select.sh | Removes obsolete “ensure xcode-select” helper script. |
| tools/devops/automation/scripts/.gitignore | Removes ignore entry for Provisionator extraction dir. |
| tools/devops/automation/run-nightly-codeql.yml | Adds arm64 agent demand for Apple-silicon-only Xcode package. |
| tools/devops/automation/build-pull-request.yml | Removes Provisionator channel parameter from PR pipeline entrypoint. |
| tools/devops/automation/build-pipeline.yml | Removes Provisionator channel parameter from CI pipeline entrypoint. |
| system-dependencies.sh | Adds --xcode-package-directory support and routes installs/verifies through install-xcode.sh. |
| Make.config | Adds pinned Xcode Universal Package coordinates (feed/name/version/build). |
| .gitignore | Stops ignoring Provisionator-generated artifacts that no longer exist. |
| .agents/skills/macios-xcode-beta-update/SKILL.md | Updates internal “Xcode beta update” skill doc for new package/build-version requirements. |
Comment on lines
+687
to
+690
| if [[ -n "$selected_changed" ]]; then | ||
| run_privileged /usr/bin/xcode-select --switch "$XCODE_APP" | ||
| install_command_line_tools | ||
| fi |
Member
Author
There was a problem hiding this comment.
Leaving this one as is — passing the .app bundle is documented usage, not a violation of the contract. From man xcode-select:
--switch path/to/Xcode.appto specify the Xcode that you wish to use for command line developer tools.
Both forms are accepted and equivalent. You are right that Makefile:77 uses xcode-select -s $(XCODE_DEVELOPER_ROOT), so the repo does use both, but neither is wrong.
There is a reason to prefer $XCODE_APP specifically here. When we reuse an Xcode that is already installed under another name, $XCODE_APP is a symlink at the Make.config path pointing at the real bundle. Selecting the symlinked app keeps xcode-select agreeing with the path every other consumer reads from Make.config, rather than pinning the resolved name.
That is also why this cannot drift: the check that decides whether to switch canonicalizes both sides before comparing —
if [[ ... || "$selected_canonical" != "$expected_canonical" || ... ]]; then
selected_changed=1
fiso it does not matter whether xcode-select reports the symlink or the resolved path; a symlinked install does not cause a re-select on every run. The CI logs bear that out — Selected Xcode 26.6.0 (17F113). appears once per provision, including on the agents that reused a differently-named bundle.
This comment has been minimized.
This comment has been minimized.
…name The comment still named download-xcode-package.sh, which stopped existing when that script became resolve-xcode-package.sh: it no longer downloads anything, the UniversalPackages task does. main has the same stale name and needs the same fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e134d13d-5eee-4266-ba75-24422addd02b
This comment has been minimized.
This comment has been minimized.
rolfbjarne
approved these changes
Jul 28, 2026
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
dalexsoto
enabled auto-merge (squash)
July 28, 2026 15:31
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Collaborator
✅ [CI Build #bcb129e] Prepare .NET Release succeeded ✅📦 Published NuGet packages (32 packages)iOS
MacCatalyst
macOS
tvOS
Other
Pipeline on Agent |
Collaborator
✅ [CI Build #b4bd7ea] Build passed (Build packages) ✅Pipeline on Agent |
Collaborator
✅ [CI Build #b4bd7ea] Build passed (Detect API changes) ✅Pipeline on Agent |
Collaborator
✅ API diff for current PR / commitNET (empty diffs)✅ API diff vs stableNET (empty diffs)ℹ️ Generator diffGenerator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes) Pipeline on Agent |
Collaborator
✅ [CI Build #b4bd7ea] Build passed (Build macOS tests) ✅Pipeline on Agent |
Collaborator
🚀 [CI Build #bcb129e] Test results 🚀Test results✅ All tests passed on VSTS: test results. 🎉 All 167 tests passed 🎉 Tests counts✅ cecil: All 1 tests passed. Html Report (VSDrops) Download macOS tests✅ Tests on macOS Monterey (12): All 5 tests passed. Html Report (VSDrops) Download Linux Build VerificationPipeline on Agent |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #26309 to
release/10.0.1xx.Cherry-picked from
fcc1dffc8417462c562c4b360d2720d5d5f1ef83with-x. See the original PR for the full rationale; this description covers only what the backport had to resolve.What this does
Replaces Provisionator with an immutable Azure Artifacts Universal Package (
xcode-apple-silicon) pinned inMake.config, downloaded by the built-inUniversalPackages@0task using the pipeline's own build identity, and installed by a repository-owned script that verifies it end to end.No Azure subscription service connection, storage account, SAS token, GitHub PAT, or Provisionator bootstrap is involved any more, and no token is handed to a script. The feed is resolved per organization —
devdiv→DevDiv,dnceng→internal.Conflicts resolved
Four, all caused by
release/10.0.1xxbeing behindmainon unrelated changes rather than by the backported work itself.tools/devops/automation/templates/common/provision.ymlandtools/devops/provision-shared.in.csx— modify/delete. Both are deleted by this change. The branch's only divergence in them was that it lacked twomain-only fixes (aretryCountOnTaskFailureon the SAS-token task, and atry/catcharoundListXcodes), and both files disappear entirely, so the deletions were taken..gitignore— the branch still carriesbcl-test-importerand the bcl-test entries thatmainhas since dropped. Kept the branch's entries and applied only this change's deletion ofprovisionator.msbuild.props.tools/devops/Makefile— this one mattered.mainhas migratedLocProject.jsontogenerate-loc-project.sh, which does not exist on this branch, so takingmain's side would have brokenmakeintools/devops. Kept the branch's existingLocProject.json.inrule and removed only theprovision-shared.csxrule and theprovisioning:target.Verification
install-xcode.sh,resolve-xcode-package.shandprovision-xcode.yml— are byte-identical tomain.main. For all nine files that still differ, the difference is exactly the divergence that already existed between the branches before the cherry-pick, compared content-line by content-line.provision.yml,provisionator-bootstrap.sh,provision-shared, the.csxscripts,provisionatorChannel,PROVISIONATOR_CHANNEL,ensure-xcode-select.sh).provision-xcode.ymlcall sites is declared, and every${{ parameters.* }}reference resolves.shellcheck0.10.0 and 0.11.0,yamllint(both the config and GitHub-Actions forms),bash -non every changed script, andtools/devops/Makefilestill parses.Make.configresolvesXCODE_PACKAGE_VERSION=26.6.0,XCODE_BUILD_VERSION=17F113and derivesXCODE_IS_STABLE=true;install-xcode.sh verifyruns; the resolver still mapsdevdiv→DevDivanddnceng→internal.The Xcode pinned here is the same as on
main(26.6/26.6.0/17F113), so no coordinate adaptation was needed.Worth watching in CI
--provision-xcodeclearsIGNORE_XCODE, socheck_xcodenow runs end to end on build bots. That newly enables theiPhoneSimulator/MacOSX/AppleTVOSSDK assertions on the build stage, which means a stale*_NUGET_OS_VERSIONinMake.versionswill fail the build stage rather than only the tests. Relevant here because this branch tracks a different set of .NET 10 package versions thanmain.