fix(security): authenticate extension services

[pimlock]

Summary

Establish the first extension-authentication contract for OpenShell. Remote supervisor middleware and gateway-interceptor RPCs receive short-lived, exact-audience gateway-signed JWTs through a shared extension client foundation.

This is intentionally an alpha integration. It wires the mechanism into middleware and interceptors now, while keeping the identity, claims, credential rotation, and transport contracts reusable by future extension points.

Related Issue

Closes #2623

Security Fix

Remote extension services previously could not cryptographically distinguish OpenShell gateway or sandbox-supervisor calls from direct network callers. This change adds authenticated OpenShell caller identity and policy-constrained token distribution while preserving the existing Ed25519 signing authority and sandbox refresh API.

These tokens identify the calling OpenShell gateway or sandbox supervisor. They do not represent an end user, delegated user authority, or durable agent identity.

Severity Assessment

• Impact is high for network-reachable extensions that perform privileged transformations or actions, and medium when trusted network isolation is enforced.
• Exploitability is moderate because exploitation requires extension endpoint reachability but not compromise of gateway signing material.
• Alpha residual risks remain: bearer replay until short expiry and single-key JWKS rotation. Multi-key overlap, mTLS, request binding or proof of possession, and stronger revocation remain deferred hardening.

Changes

• Add openshell-extension-core, a shared foundation for extension identity, claims, bearer rotation, and TLS/custom-CA transport used by both supervisor middleware and gateway interceptors.
• Make ExtensionKind non-exhaustive so future extension points can join the shared model without breaking downstream consumers. New kinds still require explicit registration, authorization, and transport wiring before OpenShell accepts them.
• Mint short-lived, exact-audience JWTs for gateway and sandbox-supervisor callers from the gateway's existing Ed25519 signing authority.
• Distribute policy-authorized credentials to supervisors through RefreshSandboxToken, held in a supervisor-owned store that rotates slots in place.
• Publish verification material at /.well-known/jwks.json and /.well-known/openid-configuration.
• Add an optional expected_audience value to Describe manifests as a post-authentication consistency assertion. Operator configuration remains authoritative, and a strict verifier may reject a wrong audience before returning the manifest.
• Add allow_insecure_transport per registration for deployments that keep a plaintext endpoint.
• Update architecture, RFC appendices, user documentation, and generated Go bindings.

Current alpha coverage

• Gateway interceptors receive gateway-caller credentials on authenticated Describe, Evaluate, and provider-profile snapshot RPCs.
• Remote supervisor middleware receives gateway-caller credentials during registration and supervisor-caller credentials during sandbox request evaluation.
• The common API covers extension kind, exact audience, caller kind, optional sandbox identity, JWT type, token lifetime and identity, rotatable bearer slots, and outbound TLS/custom-CA transport.
• Compute and credential drivers are not wired to extension JWT authentication in this PR. The shared enum is intentionally non-exhaustive, so adding those or other extension points later does not require a breaking API change.
• expected_audience does not bootstrap or discover the audience. It can detect configuration drift only after authenticated Describe succeeds.
• A token is reused across calls until rotation. jti identifies that token instance for correlation and future explicit revocation; rejecting repeated jti values would reject legitimate requests.

Review notes

Compatibility. The gateway now advertises extension-auth capability to supervisors. Without gateway_jwt, supervisors keep the legacy unauthenticated connector and never request extension credentials. With signing configured, credential or authenticated-connection failures remain fail closed. Authenticated network endpoints use https://; allow_insecure_transport = true remains the explicit per-registration plaintext opt-out.

Rotation cadence. Configuration polling runs every 10s while credentials last 15 minutes. Supervisors reuse their installed slots and rotate only when one is missing or has passed four fifths of its lifetime, so a routine poll does not re-run gateway effective-policy resolution or re-mint the gateway token.

Authorization resolution. handle_get_sandbox_config is the single resolver deciding which registrations a sandbox may hold credentials for. A cheaper parallel resolver would duplicate policy composition and backfill side effects on a security-critical path, where drift could fail open.

RFC layout. Extension-authentication detail lives in appendices/extension-authentication.md under RFC 0009 for the shared contract and RFC 0010 for interceptor differences, leaving both accepted RFC bodies intact.

Follow-ups

• Provide service-side verification SDKs, with Rust and Python first. Until then, extension authors implement the documented verification contract or use the separate authenticated middleware example as a reference.
• Wire the shared mechanism into remote driver protocols when those extension points adopt network authentication. No compute-driver or credential-driver enum entries are added before that contract exists.
• Separate extension signing authority from sandbox admission authority and support overlapping keys for zero-downtime rotation.
• Add stronger replay protection through request binding, channel binding, or proof of possession, plus explicit token revocation if required.
• Treat operator-provisioned public key or JWKS material as the authoritative cold-start trust anchor. Use the gateway well-known endpoints only for steady-state refresh after gateway identity and TLS trust are established.
• Consider mTLS client authentication, health checks, and runtime registration after the initial integration is exercised by users.

Testing

• mise run pre-commit passes, including workspace and E2E strict Clippy checks
• mise run test passes, including no-signer compatibility and signer-enabled fail-closed regression coverage
• Go protobuf bindings regenerate cleanly and mise run go:proto:check passes
• Private-CA success, wrong CA, and hostname-mismatch tests added
• Authorization, exact-audience, expiry, rotation, detachment, secret-descriptor, JWKS, and discovery-document tests added or updated
• Explicit-typing, audience-consistency, insecure-opt-out, credential-store, and mint-bound tests added
• Full Docker-backed E2E suite not run; no external extension fixture currently exists in that lane

Checklist

• Follows Conventional Commits
• Commits are signed off for DCO
• User-facing and architecture documentation updated
• Agent infrastructure inventory synchronized for the new crate

Example branch

The runnable authenticated middleware example is intentionally kept on the separate 2623-authenticated-middleware-example/pimlock branch. Its service-side JWT verifier is a dedicated example module marked for future extraction into an extension-building SDK, keeping inbound verification concerns out of openshell-extension-core. After this PR lands, the branch can be rebased and merged separately or folded into later work. Its verifier must require typ: openshell-ext+jwt, validate the configured exact audience before dispatch, and may return expected_audience from Describe as a post-authentication consistency assertion.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[pimlock]

/ok to test 4dc58b4

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-2638.docs.buildwithfern.com/openshell

[pimlock]

/ok to test 32f93e1

[lbelyaev]

On the JWKS bootstrap follow-up: operator provisioning the key seems the better default. Binding the listener early to serve only well-known paths adds a readiness state whose only job is to hand out material an out-of-band channel already has to deliver for a cold start — the first-contact service still needs something before the gateway is reachable. Treating the certgen-written key as the supported bootstrap keeps one trust path instead of two.

The well-known endpoints are still worth keeping as the steady-state convenience for services that can already reach the gateway; the question is only which one is authoritative for first contact.

[pimlock]

gator-agent

PR Review Status

Validation: Project-valid maintainer-authored implementation of linked validated issue #2623.
Head SHA: ca8cbf4667dc58b9b3ce7a65c581775baf1b815e
Base SHA: cd4d90579c542ef5b35c74eaad51acc896c53775
Merge base SHA: cd4d90579c542ef5b35c74eaad51acc896c53775
Patch ID: a39b0015ff6d36e22d1b5b15607b59ff365dd314
Gator payload: 5
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no

Blocking findings:

• GATOR-ca8cbf46-01: no-signer gateways regress from supported unauthenticated supervisor middleware to a permanently degraded registry.
• GATOR-ca8cbf46-02: the governance interceptor example has a malformed InterceptorManifest literal and cannot compile.

Carried findings:

• None

Non-blocking suggestions:

• Update debug-openshell-cluster to inspect and diagnose the new extension-authentication fields and failure modes.

Docs: Fern docs and gateway configuration references are updated for the direct UX changes.

Next state: gator:in-review

[pimlock]

@lbelyaev Agreed, and I incorporated this direction in dfe26f88: operator-provisioned public key or JWKS material is now documented as the authoritative cold-start trust anchor, while the gateway well-known endpoints are retained for steady-state refresh after gateway identity and TLS trust are established. The PR description records the same handoff decision.

[pimlock]

gator-agent

PR Review Status

Thanks @pimlock. I checked the latest author delta against the two prior findings: the advertised extension-authentication capability restores the no-signer legacy connector path, and the governance interceptor manifest literal is now well-formed with dedicated check/format coverage.

Validation: Project-valid maintainer-authored implementation of linked validated issue #2623.
Head SHA: dfe26f88282f9a14a4272a773a69303da22f05a6
Base SHA: cd4d90579c542ef5b35c74eaad51acc896c53775
Merge base SHA: cd4d90579c542ef5b35c74eaad51acc896c53775
Patch ID: def29f76fb406d6530e0e7ba7f011ac84a6367ca
Gator payload: 5
Review mode: follow_up
Previous reviewed SHA: ca8cbf4667dc58b9b3ce7a65c581775baf1b815e
Review budget exhausted: no
Maintainer decision required: no

Blocking findings:

• No blocking findings remain.

Carried findings:

• GATOR-ca8cbf46-01: resolved by the explicit gateway capability and legacy unauthenticated connector selection.
• GATOR-ca8cbf46-02: resolved by moving expected_audience inside the manifest literal and adding example checks.

Docs: Fern docs, gateway configuration references, architecture notes, and RFC appendices cover the direct UX and trust-bootstrap changes.

Test dispatch: test:e2e is required because this changes gateway/supervisor interaction and extension credential flow, but applying that label was not authorized for this cycle. No workflow was rerun and no /ok to test command was posted.

Next state: gator:blocked
Blocked reason: test_dispatch_required
Next action: the operator must separately authorize applying test:e2e; after the required workflow is actually queued, running, or complete, gator can enter pipeline watch.

[github-actions]

Label test:e2e applied for dfe26f8. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[pimlock]

gator-agent

PR Review Status

Thanks @drew, I checked the token-confusion path you identified at the sandbox authentication boundary. Thanks @pimlock, I reviewed the follow-up delta that rejects extension-typed JWTs before authenticator fallthrough, reserves the gateway sandbox audience from extension minting, and adds targeted regressions for both invariants. The concern is resolved on this head.

Validation: Project-valid maintainer-authored implementation of linked validated issue #2623.
Head SHA: 5569ccac4cd8e735f11ff8dea7f5e348c2fbc2f9
Base SHA: cd4d90579c542ef5b35c74eaad51acc896c53775
Merge base SHA: cd4d90579c542ef5b35c74eaad51acc896c53775
Patch ID: 1aa823e573b0294b3a85ea39fd96b90e04573cc1
Gator payload: 5
Review mode: follow_up
Previous reviewed SHA: dfe26f88282f9a14a4272a773a69303da22f05a6
Review budget exhausted: no
Maintainer decision required: no

Blocking findings:

• No blocking findings remain.

Carried findings:

• GATOR-ca8cbf46-01: remains resolved by the explicit gateway capability and legacy unauthenticated connector selection.
• GATOR-ca8cbf46-02: remains resolved by the corrected manifest literal and dedicated example checks.

Trusted maintainer feedback:

• @drew's sandbox-token confusion concern is resolved by the current author delta.

Docs: Fern docs, gateway configuration references, architecture notes, and RFC appendices cover the direct UX and trust-bootstrap changes.

Test dispatch: test:e2e is present, and the required current-head Branch Checks and E2E workflows are queued or running. No workflow was rerun and no /ok to test command was posted by gator.

Next state: gator:watch-pipeline

[pimlock]

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Head SHA: 5569ccac4cd8e735f11ff8dea7f5e348c2fbc2f9
Gator payload: 5

Final status: the PR reached gator:merge-ready with maintainer approval before merge.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.