-
Notifications
You must be signed in to change notification settings - Fork 200
Add bump-sdk skill #5991
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Add bump-sdk skill #5991
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,98 @@ | ||||||
| --- | ||||||
| name: bump-sdk | ||||||
| description: "Use when bumping the databricks-sdk-go dependency, upgrading the Go SDK, updating the .codegen/_openapi_sha spec SHA, or moving the CLI to the SDK version used by a given Terraform provider release." | ||||||
| user-invocable: true | ||||||
| allowed-tools: Read, Edit, Write, Bash, Glob, Grep, WebFetch, AskUserQuestion | ||||||
| --- | ||||||
|
|
||||||
| # Bump the Go SDK | ||||||
|
|
||||||
| The SDK version lives in `go.mod` (`github.com/databricks/databricks-sdk-go`) and the pinned spec SHA lives in `.codegen/_openapi_sha`. | ||||||
| These two move as a pair; everything else in this skill is regenerated from them or is fallout you fix by hand. | ||||||
| Do not hand-edit generated files (`.codegen/cli.json`, `cmd/workspace/*`, `cmd/account/*`, `bundle/schema/jsonschema.json`, `bundle/internal/validation/generated/*`, `bundle/direct/dresources/resources.generated.yml`, `bundle/terraform_dabs_map/generated.go`, `python/databricks/bundles/**`); regenerate them. | ||||||
|
|
||||||
| ## Steps | ||||||
|
|
||||||
| **1. Resolve the target versions.** | ||||||
| Use the SDK version the user gave, or if they want "the version the Terraform provider vX.Y.Z uses", read the provider's `go.mod` at that tag: `https://raw.githubusercontent.com/databricks/terraform-provider-databricks/vX.Y.Z/go.mod`. | ||||||
| The accompanying OpenAPI SHA is pinned inside the SDK at that tag: `https://raw.githubusercontent.com/databricks/databricks-sdk-go/vX.Y.Z/.codegen/_openapi_sha`. | ||||||
| Always bump the spec SHA together with the SDK; they are expected to move as a pair. | ||||||
|
|
||||||
| **2. Apply the core bump.** | ||||||
| Run `go get github.com/databricks/databricks-sdk-go@vX.Y.Z` to update `go.mod`/`go.sum`. | ||||||
| Edit `.codegen/_openapi_sha` to the new SHA; the file has no trailing newline, so match the existing format. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I've seen Claude struggle to maintain the format and go in a loop writing scripts to accomplish this - I'd say just is sufficient since the file gets auto-formatted by |
||||||
| Run `go mod tidy` and confirm the `go.mod`/`go.sum` diff is the SDK line only. | ||||||
|
|
||||||
| **3. Regenerate cli.json via genkit.** | ||||||
| Run `./task generate-clijson`, which needs a clean universe checkout at `$UNIVERSE_DIR` (default `$HOME/universe`) and builds genkit from its HEAD. | ||||||
| This task authenticates to fetch the spec by SHA, so an SSO/S3 fallback prompt during the run is normal. | ||||||
| The task also syncs `internal/genkit/tagging.py`, its two `.lock` files, and `.github/workflows/tagging.yml` from the producer HEAD; this is drift unrelated to the SDK bump. | ||||||
| Decide with the user whether to keep the tagging-producer drift or restore those files to `origin/main`. | ||||||
|
|
||||||
| **4. Regenerate everything downstream.** | ||||||
| Run `./task generate-cligen` to regenerate the command stubs from the refreshed `.codegen/cli.json`. | ||||||
| Run `./task generate-check` to regenerate the reproducible artifacts (schema, validation, direct-engine YAML, refschema, pydabs); a clean tree afterwards means zero drift. | ||||||
| Regenerate the DABs<->TF field map with `./task generate-schema-map`, which `generate-check` does not run. | ||||||
| Run `go build ./...` and fix compile breakages before touching acceptance goldens. | ||||||
|
|
||||||
| **5. Handle SDK breaking changes.** | ||||||
| Read the SDK's `CHANGELOG.md` at the target version (in the module cache) to enumerate breaking changes before chasing compile errors. | ||||||
| A removed struct field that the CLI used (e.g. `jobs.AiRuntimeTask.CodeSourcePath`) should have its usage temporarily disabled with a comment noting it returns in a later SDK bump, not deleted outright. | ||||||
| A new struct field triggers an `exhaustruct` lint failure in `bundle/direct/dresources/*`; wire the field through `PrepareState` and `RemapState` when it exists on both the input and remote types. | ||||||
| A field the new spec now annotates as output-only may already be emitted into `resources.generated.yml`, making the manual entry in `resources.yml` redundant; `TestResourcesYMLNoRedundantRules` catches this, so remove the manual entry. | ||||||
|
|
||||||
| **6. Refresh goldens, then VERIFY.** | ||||||
|
|
||||||
| ```bash | ||||||
| go test ./acceptance -run '^TestAccept$' -update -timeout=60m | ||||||
| go test ./acceptance -run '^TestAccept$' -timeout=60m # MUST pass on its own | ||||||
| ``` | ||||||
|
|
||||||
| The verify pass is not optional. | ||||||
| Bundle tests run under an `EnvMatrix` of both engines (`terraform`, `direct`). | ||||||
| When a schema change makes one engine error while the other succeeds, the variants produce different output; `-update` runs both and each overwrites the other's `output.txt`, so it can silently settle on the passing variant and report `ok` while the golden is actually wrong. | ||||||
| Only the non-update run catches this. | ||||||
| (Ignore `rejecting_proxy.go: blocking proxy` log lines, which are normal. | ||||||
| A test that times out under full parallel load but passes when run alone is a flake, not a regression.) | ||||||
|
|
||||||
| **7. Handle acceptance-test fallout** if the verify pass fails. | ||||||
| A field that becomes required in the spec makes the CLI emit a client-side "required field ... is not set" warning wherever a test config omits it. | ||||||
| For a test that legitimately sets the field, add a non-empty value; if a sibling Terraform-provider bump PR is open, reuse its exact value to avoid conflicts. | ||||||
| For a test that deliberately omits the field to exercise a backend error, that test is likely now redundant with generic client-side validation, so confirm with the user and `git rm` it. | ||||||
| Regenerate the affected test's `out*` files with `go test ./acceptance -run 'TestAccept/<path>' -update` (never hand-edit them), then re-run the full verify pass. | ||||||
|
|
||||||
| **8. Verify the rest.** | ||||||
| Run `./task fmt` and `./task lint-q`; if either touches `acceptance/`, a fixture is wrong, so fix the source rather than editing output. | ||||||
| Run the full root-module unit suite: `go test ./acceptance/internal ./libs/... ./internal/... ./cmd/... ./bundle/... ./experimental/... .`. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. use the |
||||||
| If a test fails, confirm whether it is pre-existing by reproducing it on the base commit in a throwaway worktree (`git worktree add --detach /tmp/base HEAD`) before assuming the bump caused it. | ||||||
| Confirm no internal proxy URL leaked into any lock file: `./task check-uv-lock` covers `*uv.lock`, but the genkit `*.py.lock` files are outside that glob, so grep them for `pypi-proxy.cloud.databricks.com` separately. | ||||||
| The `check-uv-lock` glob and the genkit lock revert are a known coverage gap; the internal proxy can re-leak into `internal/genkit/*.py.lock` on any future `generate-clijson`, so re-check after every run. | ||||||
|
Comment on lines
+68
to
+69
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. note to self: need to wrap up #5830 |
||||||
|
|
||||||
| **9. Changelog fragment.** | ||||||
| Add a `dependency-updates` entry per the `pr-checklist` skill's "Changelog entry" section, modeled on prior bumps: ``Bump `github.com/databricks/databricks-sdk-go` from vOLD to vNEW.``. | ||||||
| Never reference the Terraform provider version in the changelog fragment or PR body. | ||||||
| Add it without `(#NNNN)` now; backfill the number after the PR exists, then run `./task links` to expand it into the full markdown link in place and commit the result. | ||||||
|
|
||||||
| **10. Commit, push, PR.** | ||||||
| If the push 403s, the active gh account lacks write access to `databricks/cli`; switch to one that has it with `gh auth switch`. | ||||||
| Then follow the `pr-checklist` skill for the PR, and do not run `gh pr create` without the user's explicit permission. | ||||||
| Commit body and PR description: | ||||||
|
Comment on lines
+74
to
+79
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we should consolidate into |
||||||
|
|
||||||
| ``` | ||||||
| ## Changes | ||||||
|
|
||||||
| Bump `github.com/databricks/databricks-sdk-go` from v{old_version} to v{version}. | ||||||
|
|
||||||
| Notable SDK/spec changes: | ||||||
| - <one terse bullet per user-visible breaking change or new field handled> | ||||||
|
|
||||||
| ## Tests | ||||||
|
|
||||||
| Generated artifacts and acceptance goldens regenerated; full unit and acceptance suites pass. | ||||||
| ``` | ||||||
|
|
||||||
| ## Stacking and rebasing | ||||||
|
|
||||||
| If a Terraform-provider bump PR is open and touches the same files (the bind-test `role` value and `bundle/terraform_dabs_map/generated.go`), stack on it or merge it once landed. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
too specific |
||||||
| On any merge/rebase conflict in a generated file like `generated.go`, resolve to a compilable state and then regenerate rather than hand-merging. | ||||||
| A squash-merged upstream PR leaves its original commits as non-ancestors, so the branch log may show duplicate commits even when the net diff vs `origin/main` is correct. | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the skill should look up the latest if none provided - now the step sounds like user input is required so it will ask