refactor(compute): define driver shutdown and restart lifecycle contract

[elezar]

Problem Statement

The OpenShell Drivers roadmap (#1051) requires in-tree and out-of-tree compute drivers to interact with the gateway through the same gRPC contract, but driver shutdown and restart semantics are not defined by compute_driver.proto.

The gateway currently has different lifecycle ownership for in-process drivers, the gateway-managed VM driver process, and operator-managed external driver endpoints. Without a shared contract, graceful gateway shutdown, driver restart, sandbox recovery, and reconnection behavior remain driver-specific and cannot be covered consistently by conformance tests.

Proposed Design

Define a common driver lifecycle contract before adding implementation-specific hooks. The design should specify:

• which component owns starting, stopping, and restarting each driver process;
• whether the protocol needs an explicit graceful-shutdown RPC, a drain or quiesce operation, capability negotiation, or a combination;
• what a successful shutdown response guarantees for running sandboxes and watch streams;
• how the gateway detects driver loss and reconnects after a driver restart;
• how persisted sandboxes are reconciled after gateway or driver restart;
• how unmanaged out-of-tree drivers express unsupported lifecycle operations;
• timeout, idempotency, and error semantics; and
• which behaviors belong in the driver conformance suite (spike: driver conformance suite for compute drivers #2183).

Keep process supervision separate from the protocol. Package managers or operators may own an external process even when the gateway uses a common lifecycle RPC.

Alternatives Considered

• Continue relying on process termination and reconnect behavior only. This leaves graceful drain and recovery semantics implicit and driver-specific.
• Make the gateway supervise every driver process. This does not fit operator-managed out-of-tree endpoints.
• Add a shutdown RPC without first defining its guarantees. This risks encoding incompatible behavior across drivers.

Agent Investigation

• The roadmap lists "Shutdown and restart mechanics" as remaining work but links no issue or PR.
• Repository searches for driver shutdown and restart lifecycle work found no dedicated contract tracker.
• Spike: should the built-in compute drivers move to the external (out-of-process) model? #2061 investigates whether built-in drivers should move out of process, but deliberately does not define the lifecycle protocol.
• Sandboxes stuck in Error after Podman machine restart — no recovery path #2179 and PR fix(compute): recover Error-phase sandboxes on gateway startup #2269 cover a concrete Podman/Docker restart-recovery failure and should inform the common contract.
• spike: driver conformance suite for compute drivers #2183 defines the driver conformance direction and should consume the lifecycle contract once agreed.
• refactor(compute): define a uniform gateway callback endpoint contract #2215 provides a pattern for defining a cross-driver semantic contract before selecting a protobuf shape.

Definition of Done

• Document lifecycle ownership for in-process, gateway-managed, and operator-managed drivers.
• Decide whether new protocol methods or capabilities are required.
• Define shutdown, disconnect, restart, reconnect, and reconciliation semantics.
• Identify migration work for all in-tree drivers and external drivers.
• Add or update implementation issues and conformance scenarios.
• Update the relevant architecture documentation.

[github-actions]

This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

[drew]

🏗️ build-plan

Implementation Plan

Issue type: refactor
Complexity: Medium
Confidence: High — clear path

Summary

Remove the Docker-only gateway shutdown and startup hooks. Gateway shutdown will no longer proactively stop Docker sandboxes; on startup, the gateway will reconcile persisted running intent through the shared idempotent ComputeDriver::StartSandbox RPC added by #2653. A stacked follow-up will remove the remaining server-side behavioral checks for concrete driver names: lifecycle reconciliation and process-identity defaults will be negotiated as driver capabilities, listener requirements will be validated generically, and canonical built-in names may be supplied through the same operator-owned UDS endpoint as extension drivers.

Scope

• Keep refactor(compute): unify gateway restart reconciliation #2743 focused on removing Docker-only shutdown behavior and adding shared startup reconciliation.
• Extend compute_driver.proto with additive, forward-compatible feature negotiation for startup reconciliation and unspecified process-identity handling.
• Replace runtime behavior gates based on docker, podman, kubernetes, or vm names with negotiated features or generic structural validation.
• Allow --drivers <canonical-name> --compute-driver-socket <path> to select a remote driver; the endpoint override takes precedence over built-in construction.
• Prove parity with a fake UDS driver using a canonical name and update the public extension-driver contract documentation.
• Do not change default packaging, spawn additional driver subprocesses, or remove in-process built-in constructors. That migration remains separate from refactor(compute): define driver shutdown and restart lifecycle contract #2417.

Implementation Steps

1. Land refactor(compute): unify gateway restart reconciliation #2743's shared lifecycle path: remove Docker shutdown cleanup and reconcile persisted local running intent through StartSandbox.
2. Add a ComputeDriverFeature enum to GetCapabilitiesResponse. Unknown and absent features are ignored so existing extension drivers keep conservative behavior.
3. Advertise startup reconciliation from Docker, Podman, and VM; advertise unspecified process-identity preservation from Docker and Podman. Kubernetes advertises neither.
4. Snapshot negotiated features in ComputeRuntime; use them for startup reconciliation and policy normalization. Keep driver kind only for configuration and telemetry.
5. Remove the gateway-listener driver-name allowlist while retaining callback authorization and existing address, multicast, wildcard, and port validation.
6. Permit canonical built-in names with --compute-driver-socket; without an endpoint, keep existing in-process construction.
7. Update architecture and user documentation to define compute_driver.proto as the supported gateway/driver extension boundary and explain endpoint precedence.

Test Plan

• Unit-test feature advertisement for Docker, Podman, Kubernetes, and VM.
• Test startup reconciliation and process-identity behavior by feature, including an arbitrary or canonical name over the fake UDS driver and conservative behavior with no feature.
• Test listener requirements independently of driver name while preserving unsafe-address and port rejection.
• Test CLI/config resolution: a canonical name plus endpoint resolves remotely; the same name without endpoint remains built in; endpoint selection still requires exactly one explicit driver.
• Run focused crate tests, mise run pre-commit, mise run test, and mise run ci. Retain refactor(compute): unify gateway restart reconciliation #2743's Docker restart E2E as the lifecycle regression.

Risks & Open Questions

• Legacy external drivers advertise no features and therefore retain today's conservative behavior.
• Advertised driver name is informational, not an authorization boundary; socket filesystem permissions and callback-scoped RPC authorization remain the trust boundaries.
• Listener requirements remain structurally constrained even though they are no longer gated by configured driver name.
• Actual standalone Docker packaging, UDS modes for other built-ins, gateway-managed launching, and removing concrete server dependencies require separate acceptance/design under Spike: should the built-in compute drivers move to the external (out-of-process) model? #2061.

Documentation Impact

• Update architecture/compute-runtimes.md, docs/reference/sandbox-compute-drivers.mdx, and docs/reference/gateway-config.mdx for the negotiated feature and canonical endpoint behavior.
• Review the cluster-debug skill for stale reserved-name diagnostics.
• LSM compatibility is unchanged: this changes metadata negotiation, not process visibility or execution controls.

---

Revision 2 — adds the stacked capability-driven external-driver parity layer after #2743

[drew]

Implementation is ready in #2743.

This removes the Docker-specific gateway shutdown cleanup and uses the shared ComputeDriver::StartSandbox contract to reconcile persisted running intent for Docker, Podman, and VM sandboxes at gateway startup. Explicitly stopped sandboxes remain stopped; Kubernetes keeps its cluster-owned lifecycle.

Local verification:

• mise run pre-commit passed
• startup reconciliation unit tests passed
• Docker/Podman/VM E2E targets compile
• Docker restart E2E passed with running + explicitly stopped sandboxes
• Podman runtime E2E unavailable on this host (no Podman CLI)
• VM lane was blocked by its pre-existing smoke readiness failure before the restart scenario ran

[drew]

Implementation is ready in stacked PR #2744, based on #2743. This follow-up replaces post-initialization driver-name behavior gates with additive capability negotiation, permits canonical driver names over the public UDS contract, and tests Docker-compatible lifecycle/policy behavior through a fake external driver. Merge order: #2743, then #2744.