Skip to content

docs: list the 2026-11-10 index builds and migrations with measured times [ENG-2184] #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
twk3 wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

docs: list the 2026-11-10 index builds and migrations with measured times [ENG-2184] #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions docs/upgrading.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,39 @@ Check running container versions:
docker compose ps --format "table {{.Name}}\t{{.Image}}"
```

## Upgrading from 2026-10-14 to 2026-11-10

`2026-11-10` builds six MongoDB indexes and runs five migrations when the scheduler starts. None of the indexes is on `instances`, `runs` or `testRecord`, and the migrations read only keys and secrets, so there is nothing to build ahead of the upgrade. On a test install with 3.1 million `instances` (41 GB), 310,000 runs and 124,000 orchestration sessions, the scheduler logged `Startup tasks complete` 18 seconds after `docker compose up -d`.

Run the check in [Index Builds Before an Upgrade](#index-builds-before-an-upgrade) before `docker compose up -d`. On the test install it exited `0` with an estimate of 25 seconds, and the builds took 1 second.

Of this release's indexes, only the one on `orchestrationSession` grows with your history, and only if you use orchestration. At the check's default rate the collection would need about 18 GB to pass the 10-minute limit. If the check exits `2` and names `orchestrationSession`, build it first as in [Building the Indexes First](#building-the-indexes-first).

### What the scheduler builds and runs

| Index | Collection |
|---|---|
| `orchestrationSession` `{ runId: 1 }` | Grows with orchestrated runs. 0.79 GB took about 1 second in testing. |
| `oauthRefreshToken` `{ rotationReplayExpiresAt: 1 }` | Small: refresh tokens of MCP clients. |
| `orgStorageDaily` `{ date: 1, orgId: 1, bytes: 1 }` | Empty on-prem. |
| `scimPendingUsers` `{ orgId: 1, email: 1 }` (unique) | Written only by SCIM provisioning. |
| `customerRequests`, two indexes | Empty on-prem. |

| Migration | What it does | Time in testing |
|---|---|---|
| `20261028-drop-superseded-instances-indexes` | Drops two `instances` indexes that newer ones replace, once the replacements exist. `2026-10-14` already dropped them, so it finds nothing. | 3 ms |
| `20261103-redact-audit-log-secrets` | Masks credentials in stored audit log records. On-prem doesn't store audit records in MongoDB, so it usually finds none. | 553 ms for 10,000 records |
| `20261109-backfill-api-key-hashes` | Stores a hash and the last 4 characters of each organization API key. | 202 ms for 3,000 keys |
| `20261110-backfill-record-key-hashes` | Stores a hash and the last 4 characters of each record key. | 25 ms for 400 keys |
| `20261110-backfill-scim-secret-hashes` | Stores a hash and the last 4 characters of the SCIM secret. | 2 ms |

The two key migrations leave a key without a hash when another document holds the same value, and log a warning with the documents' ids:

- `left N organization API key(s) in M group(s) without a hash`: API keys that share a value already fail to authenticate. Delete the extra copies.
- `left N record key(s) without a hash`: a record key shared by two organizations. Have all but one of them rotate their record key; deleting a copy breaks that organization's CI.

Still on `2026-07-26`? The scheduler also builds the `2026-10-14` indexes on `instances`, `runs` and `testRecord`: estimate and build them first as in [Upgrading from 2026-07-26 to 2026-10-14](#upgrading-from-2026-07-26-to-2026-10-14). `2026-11-10` builds the replacement `instances` indexes before it drops the old ones, so reading a run's spec files keeps an index throughout.

## Upgrading from 2026-07-26 to 2026-10-14

Upgrade to `2026-10-14-002` or later. On `2026-10-14-001`, users who signed in before the upgrade can't sign in again: SAML SSO returns them to the sign-in page with `account_not_linked`, and the root admin's password sign-in answers `INVALID_EMAIL_OR_PASSWORD`. `2026-10-14-002` signs them in with no change to the database.
Expand Down Expand Up @@ -242,6 +275,8 @@ git checkout <previous-tag-or-commit>
docker compose up -d
```

Rolling back from `2026-11-10` to `2026-10-14` builds and drops nothing: the `2026-10-14` scheduler keeps the new indexes, and the stored hashes are fields it doesn't read.

Rolling back from `2026-10-14` to `2026-07-26` rebuilds indexes. When the `2026-07-26` scheduler starts, it drops the two `instances` indexes `2026-10-14` builds and rebuilds its own, which takes as long as the `instances` builds in the [estimate](#estimate-the-upgrade-time). Reading a run's spec files has no index until the first one finishes, so roll back in a maintenance window too.

## Troubleshooting
Expand Down
4 changes: 4 additions & 0 deletions on-prem/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [Unreleased]

### Upgrade Notes
- **The scheduler builds six MongoDB indexes and runs five migrations when it starts, which takes seconds.** None of the indexes is on `instances`, `runs` or `testRecord`. On a test install with 3.1 million `instances`, startup finished 18 seconds after `docker compose up -d`. Run the index check in the upgrade guide before `docker compose up -d`. See [Upgrading from 2026-10-14 to 2026-11-10](../docs/upgrading.md#upgrading-from-2026-10-14-to-2026-11-10).
- **Organization API keys, record keys and the SCIM secret get a stored hash.** A key whose value another document also holds is left without one, and the scheduler logs a warning with the documents' ids. The upgrade guide says how to resolve them.

## [2026-10-14-002] - 2026-10-27

Image-only update on `2026-10-14-001`: no compose file or environment variable changes.
Expand Down