Skip to content

Support SPIFFE-backed provider token grants on non-Kubernetes drivers (Docker, Podman, VM) #2708

Description

@yucliang-nv

Problem Statement

Provider token grants (#1784) give a sandbox a strong, per-sandbox workload identity: the supervisor fetches a JWT-SVID over the SPIFFE Workload API and exchanges it (RFC 7523 client-assertion flow) for an upstream access token, injected only at egress so the agent and sandbox never hold the credential. This is the right foundation for brokering credentials to sandboxed agents.

Today that capability is structurally Kubernetes-only, even though nothing in the consuming code is Kubernetes-specific:

  • openshell-supervisor-network/src/token_grant.rs talks to the standard SPIFFE Workload API via the spiffe crate (WorkloadApiClient) — it only needs OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET to point at a socket.
  • But the only thing that ever sets that env var is the Kubernetes driver, and the only config surface is KubernetesComputeConfig.provider_spiffe_workload_api_socket_path (crates/openshell-driver-kubernetes/src/config.rs:296).
  • The socket volume source is hardcoded to the SPIFFE CSI driver ("driver": "csi.spiffe.io", crates/openshell-driver-kubernetes/src/driver.rs:2633).
  • The Docker, Podman, and VM drivers have no SPIFFE plumbing at all (git grep -i spiffe over those crates returns nothing).

Operators run OpenShell sandboxes on standalone Linux hosts, developer workstations, and VM fleets — environments that commonly already run a SPIFFE Workload API implementation on the host (a SPIRE agent on VMs/bare metal, Teleport tbot, etc.). Those deployments currently get no per-sandbox workload identity and no token grants, and fall back to long-lived static credentials in provider profiles — exactly what token grants were built to eliminate. We would like to adopt token grants uniformly across Kubernetes, Docker, and VM sandboxes; this issue is the "report back gaps" follow-up to #1784 from trying that.

Proposed Design

Make the Workload API endpoint a first-class, driver-agnostic sandbox capability; each driver plumbs it in the way native to its platform. The supervisor-side contract (OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET, drained from the agent environment) stays unchanged.

  1. Shared config key. Lift provider_spiffe_workload_api_socket_path (or a provider_spiffe { workload_api_endpoint = … } block) out of KubernetesComputeConfig into common driver configuration, so Docker/Podman/VM accept it with identical semantics: empty = disabled.
  2. Docker/Podman drivers. Bind-mount the host socket directory read-only into the sandbox at a reserved path and set the env var for the supervisor. Preserve the existing security invariant: the mount must be visible to the supervisor only, hidden from the agent process (same property the Kubernetes driver enforces by stripping the volume from user containers and draining the env var).
  3. VM driver. Accept an in-guest endpoint path (socket provided by the guest image or forwarded from the host, e.g. over vsock); the driver's job is only to validate and expose it to the supervisor.
  4. Kubernetes driver: non-CSI socket sources. Allow a hostPath socket directory (the common spire-agent DaemonSet layout) or a pre-existing volume reference as alternatives to the hardcoded csi.spiffe.io volume source.
  5. Honor endpoint schemes that already parse. openshell-supervisor-network/src/spiffe_endpoint.rs accepts unix: and tcp: forms, but the config validator (config.rs:478) only admits absolute UNIX paths, so tcp: Workload API endpoints (used by some VM setups) are unreachable. Relax validation to admit the forms the endpoint parser supports, keeping the dedicated-directory rule for UNIX paths.
  6. Per-sandbox identity mapping guidance. On Kubernetes, per-sandbox SPIFFE IDs come from a ClusterSPIFFEID templated on the openshell.io/sandbox-id pod annotation. Off Kubernetes the equivalent needs a documented pattern; candidates, in rough order of preference:
    • the driver exposes sandbox identity attributes (sandbox id, supervisor PID/UID) that host Workload API implementations can use in workload attestation (e.g. Unix attestor);
    • one Workload API endpoint per sandbox, provisioned by the operator, with the socket path templated on sandbox id;
    • fallback: a shared host identity plus OpenShell's existing gateway-issued sandbox JWT for per-sandbox granularity.
      The design should pick one primary pattern per driver and document it; we're happy to contribute the Docker driver implementation first.

Anything that speaks the SPIFFE Workload API (SPIRE agent, Teleport tbot, cert-manager csi-driver-spiffe, …) should work without OpenShell knowing which one it is.

Alternatives Considered

  • Third-party compute drivers (RFC 0001). Re-implementing a whole compute driver to add one mount + one env var duplicates first-party drivers for a capability that is already driver-agnostic in token_grant.rs. Parity in the first-party drivers is a small, contained change; the driver contract doesn't currently carry a "Workload API endpoint" concept to delegate to third parties anyway.
  • Keep using the gateway-issued sandbox JWT off Kubernetes. It uniquely identifies the sandbox, but it's OpenShell-proprietary: upstream IdPs/OAuth2 servers can't consume it the way they consume JWT-SVIDs from a SPIFFE trust domain with published JWKS, and it doesn't federate into existing SPIFFE ecosystems.
  • Run the SPIFFE agent inside each sandbox image. Works, but requires custom sandbox images, puts an agent-adjacent process and its join material inside the workload it is supposed to attest, and every operator reinvents it per environment. Host-level Workload API + a driver mount is the standard SPIFFE deployment shape.
  • Refresh static provider credentials from an external process. Keeps egress-time injection but reintroduces out-of-band credential writes with no attestation binding — the problem token grants exist to solve.

Agent Investigation

Codebase findings, all at origin/main (0310cbed, 2026-08-11):

  • crates/openshell-supervisor-network/src/token_grant.rs:44 uses spiffe::WorkloadApiClient; :32 documents the OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET requirement; the exchange is standard RFC 7523 (urn:ietf:params:oauth:client-assertion-type:jwt-bearer, :62) — no Kubernetes dependency in this crate.
  • crates/openshell-core/src/sandbox_env.rs:101 defines the env key; the only writer is the Kubernetes driver (crates/openshell-driver-kubernetes/src/main.rs:116 and driver.rs env wiring).
  • crates/openshell-driver-kubernetes/src/config.rs:220-296, 388-508 — config field + validation (absolute UNIX path below a dedicated directory; /run, /var, /tmp, /etc rejected per docs/reference/gateway-config.mdx).
  • crates/openshell-driver-kubernetes/src/driver.rs:2633 — volume source hardcoded to csi.spiffe.io; reserved volume name + protected mount path; sidecar transform strips the mount from user containers.
  • crates/openshell-supervisor-network/src/spiffe_endpoint.rs:10-17 — already parses unix: and tcp: schemes, unreachable through current validation.
  • No SPIFFE references in crates/openshell-driver-docker/, crates/openshell-driver-podman/, crates/openshell-driver-vm/, and no SPIFFE e2e coverage under e2e/ or CI workflows; the only end-to-end artifact is the manual examples/spiffe-token-grant-demo/.
  • Helm: deploy/helm/openshell/values.yaml server.providerTokenGrants.spiffe.* renders only into [openshell.drivers.kubernetes] (templates/gateway-config.yaml:135).

Checklist

  • I've reviewed existing issues and the architecture docs
  • This is a design proposal, not a "please build this" request

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions