feat(kubernetes): add sandbox disruption protection

.agents/skills/debug-openshell-cluster/SKILL.md

@@ -362,16 +362,50 @@ helm -n openshell get values openshell | grep sandboxNamespace
 
 Then inspect sandbox resources in that namespace.
 
+For a sandbox that requested disruption protection, verify the operator gate,
+the persisted deadline, PDB ownership, and gateway RBAC:
+
+```bash
+helm -n <gateway-namespace> get values <release-name> | grep -A4 disruptionProtection
+kubectl -n <gateway-namespace> get configmap <gateway-configmap> -o jsonpath='{.data.gateway\.toml}' | grep -A3 disruption_protection
+PDB_RESOURCE=$(kubectl -n <sandbox-namespace> get sandboxes.agents.x-k8s.io -l 'openshell.ai/sandbox-name=<sandbox-name>,openshell.ai/sandbox-workspace=<workspace>' -o jsonpath='{.items[0].metadata.name}')
+kubectl -n <sandbox-namespace> get sandbox "$PDB_RESOURCE" -o go-template='{{ index .metadata.annotations "openshell.io/disruption-protected-until" }}{{ "\n" }}'
+kubectl -n <sandbox-namespace> get poddisruptionbudget "$PDB_RESOURCE" -o yaml
+for verb in create delete get list patch; do
+  kubectl auth can-i --as=system:serviceaccount:<gateway-namespace>:<gateway-service-account> "$verb" poddisruptionbudgets.policy -n <sandbox-namespace>
+done
+```
+
+Both opt-ins are required: Helm
+`server.disruptionProtection.enabled=true` and sandbox driver config
+`kubernetes.disruption_protection.duration`. A request over `maxDuration` is
+rejected. The PDB should use `minAvailable: 1`,
+`unhealthyPodEvictionPolicy: AlwaysAllow`, and an owner reference to the
+Sandbox CR. OpenShell stores an absolute UTC deadline in
+`openshell.io/disruption-protected-until`; an expired PDB can remain while all
+gateways are down but should be removed after gateway reconciliation resumes.
+PDBs apply only to voluntary Eviction API operations, not node failure,
+preemption, direct deletion, backup, or restore.
+
+Expiration is fail-closed. If every gateway is unavailable and an expired PDB
+is blocking emergency maintenance, verify the persisted deadline first, then
+remove that exact PDB manually:
+
+```bash
+kubectl -n <sandbox-namespace> delete poddisruptionbudget "$PDB_RESOURCE"
+```
+
 Check the configured sandbox service account when TokenReview bootstrap or
 sandbox registration fails. Helm creates a dedicated sandbox service account by
 default and writes it to `[openshell.drivers.kubernetes].service_account_name`;
 the gateway rejects projected tokens from other service accounts.
 
 ```bash
-helm -n openshell get values openshell | grep -A3 sandboxServiceAccount
-kubectl -n <sandbox-namespace> get serviceaccount openshell-sandbox
-kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}'
-kubectl -n <sandbox-namespace> get sandbox <sandbox-name> -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'
+helm -n <gateway-namespace> get values <release-name> | grep -A3 sandboxServiceAccount
+kubectl -n <sandbox-namespace> get serviceaccount <sandbox-service-account>
+kubectl -n <gateway-namespace> get configmap <gateway-configmap> -o jsonpath='{.data.gateway\.toml}'
+SANDBOX_RESOURCE=$(kubectl -n <sandbox-namespace> get sandboxes.agents.x-k8s.io -l 'openshell.ai/sandbox-name=<sandbox-name>,openshell.ai/sandbox-workspace=<workspace>' -o jsonpath='{.items[0].metadata.name}')
+kubectl -n <sandbox-namespace> get sandboxes.agents.x-k8s.io "$SANDBOX_RESOURCE" -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'
 ```
 
 If `topology = "sidecar"` is rendered under `[openshell.drivers.kubernetes]`,
@@ -453,6 +487,8 @@ openshell logs <sandbox-name>
 | Docker GPU e2e fails before GPU sandbox comparison | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` |
 | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod <pod>` |
 | Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` |
+| Kubernetes sandbox creation rejects a disruption-protection request | Operator gate is disabled, duration is invalid, or request exceeds `maxDuration` | CLI error, Helm values, rendered `gateway.toml`, gateway logs |
+| Existing Kubernetes sandbox requested disruption protection but has no PDB | Gateway lacks `poddisruptionbudgets.policy` RBAC or reconciliation failed | Gateway logs, Sandbox deadline annotation, `kubectl auth can-i`, PDB selector and owner reference |
 | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` |
 | CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways/<name>/mtls/` |
 | Edge or OIDC gateway returns `Unauthenticated` | Stored login expired, audience/scopes mismatch, or gateway auth configuration changed | `openshell gateway info`, `openshell gateway login <name>`, gateway auth logs |

architecture/compute-runtimes.md

@@ -282,6 +282,12 @@ workload is a StatefulSet for SQLite-backed single-replica installs. External
 database-backed installs can render a Deployment with `workload.kind=deployment`;
 HA deployments must point `server.externalDbSecret` at an operator-managed
 PostgreSQL database.
+The Kubernetes driver can also own an opt-in PodDisruptionBudget with a
+fail-closed expiration target for a sandbox. The gateway persists the absolute
+deadline on the Agent Sandbox CR, schedules deadline cleanup while its watch is
+healthy, and reconciles the PDB after interruptions. An owner reference
+provides deletion cleanup. This is voluntary-eviction protection, not sandbox
+state persistence or recovery.
 Standalone local deployments start the gateway with a selected runtime such as
 Docker, Podman, or VM. The CLI can register multiple gateways and switch between
 them without changing the sandbox architecture.

crates/openshell-driver-kubernetes/Cargo.toml

@@ -29,6 +29,7 @@ kube-runtime = { workspace = true }
 k8s-openapi = { workspace = true }
 serde = { workspace = true }
 serde_json = { workspace = true }
+chrono = { version = "0.4", default-features = false, features = ["clock", "std"] }
 clap = { workspace = true }
 tracing = { workspace = true }
 tracing-subscriber = { workspace = true }

crates/openshell-driver-kubernetes/README.md

@@ -26,6 +26,35 @@ by the gateway.
 Kubernetes API calls use explicit timeouts so gRPC handlers do not block
 indefinitely when the API server is slow or unavailable.
 
+## Disruption Protection
+
+The driver can create a `policy/v1` `PodDisruptionBudget` with a fail-closed
+expiration target for an individual sandbox. This capability uses two explicit
+opt-ins: the operator sets `disruption_protection.enabled = true`, and the
+sandbox requests `driver_config.kubernetes.disruption_protection.duration`.
+The operator's `max_duration` limits each request. Durations are positive
+integer seconds, minutes, or hours, such as `30m` or `4h`.
+
+OpenShell calculates an absolute UTC deadline and stores it in the
+`openshell.io/disruption-protected-until` annotation on both the Sandbox and
+PDB. The PDB uses `minAvailable: 1`, selects only the corresponding OpenShell
+sandbox pod, and sets `unhealthyPodEvictionPolicy: AlwaysAllow`. The driver
+creates the PDB before the Sandbox, adds the Sandbox owner reference after the
+CR exists, repairs missing or changed managed PDBs during gateway
+reconciliation, and schedules cleanup from Sandbox watch events at the
+deadline. The periodic gateway reconciliation sweep is the repair fallback.
+Expiration is fail-closed: if all gateways are unavailable at the deadline, the
+PDB remains active until a gateway resumes or an operator deletes it. A
+malformed persisted deadline also retains protection until an operator repairs
+or removes the annotation; it never serves as evidence that protection expired.
+
+PDBs constrain voluntary eviction through the Kubernetes Eviction API. They do
+not prevent node failure, preemption, direct pod or Sandbox deletion, or other
+involuntary disruption, and they do not provide sandbox backup or restore.
+The standalone driver accepts the same operator settings through
+`OPENSHELL_K8S_DISRUPTION_PROTECTION_ENABLED` and
+`OPENSHELL_K8S_DISRUPTION_PROTECTION_MAX_DURATION`.
+
 ## Workspace Persistence
 
 Sandbox pods use a PVC-backed `/sandbox` workspace. An init container seeds the
@@ -134,6 +163,7 @@ nested schema and currently accepts:
 - `volumes[].name`
 - `volumes[].persistent_volume_claim.claim_name`
 - `volumes[].persistent_volume_claim.read_only`
+- `disruption_protection.duration`
 
 Nested keys inside the `kubernetes` block use snake_case. The top-level
 `driver_config` envelope is keyed by driver names, so `kubernetes` is not part
@@ -148,6 +178,14 @@ openshell sandbox create \
   -- claude
 ```
 
+When the operator enables disruption protection, request it for one sandbox:
+
+```shell
+openshell sandbox create \
+  --driver-config-json '{"kubernetes":{"disruption_protection":{"duration":"4h"}}}' \
+  -- claude
+```
+
 Resource keys use native Kubernetes resource names and quantity strings. The
 parser renders the keys listed above and rejects unknown fields.
 `pod.runtime_class_name` maps to PodSpec `runtimeClassName` and overrides the

crates/openshell-driver-kubernetes/src/config.rs

@@ -5,6 +5,7 @@ use openshell_core::config;
 use serde::{Deserialize, Deserializer, Serialize};
 use std::path::Path;
 use std::str::FromStr;
+use std::time::Duration;
 
 /// Default Kubernetes namespace for sandbox resources.
 pub const DEFAULT_K8S_NAMESPACE: &str = "openshell";
@@ -18,6 +19,62 @@ pub const DEFAULT_WORKSPACE_STORAGE_SIZE: &str = "2Gi";
 /// Default non-root UID for relaxed Kubernetes network supervisor sidecars.
 pub const DEFAULT_PROXY_UID: u32 = 1337;
 
+/// Maximum disruption-protection duration accepted when the operator enables
+/// per-sandbox `PodDisruptionBudgets`.
+pub const DEFAULT_DISRUPTION_PROTECTION_MAX_DURATION: &str = "24h";
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(default, deny_unknown_fields)]
+pub struct KubernetesDisruptionProtectionConfig {
+    /// Allow callers to request a time-bounded `PodDisruptionBudget` for an
+    /// individual Kubernetes sandbox. Enabling this capability alone does not
+    /// protect any sandbox; the caller must also provide a duration in the
+    /// per-sandbox Kubernetes driver config.
+    pub enabled: bool,
+    /// Upper bound for a caller-provided protection duration.
+    pub max_duration: String,
+}
+
+impl Default for KubernetesDisruptionProtectionConfig {
+    fn default() -> Self {
+        Self {
+            enabled: false,
+            max_duration: DEFAULT_DISRUPTION_PROTECTION_MAX_DURATION.to_string(),
+        }
+    }
+}
+
+pub(crate) fn parse_disruption_protection_duration(value: &str) -> Result<Duration, String> {
+    let value = value.trim();
+    if value.is_empty() {
+        return Err("duration must not be empty".to_string());
+    }
+
+    let unit_len = value.chars().last().map_or(0, char::len_utf8);
+    let (amount, unit) = value.split_at(value.len() - unit_len);
+    let amount = amount.parse::<u64>().map_err(|_| {
+        format!("invalid duration '{value}'; expected values such as 30m, 4h, or 24h")
+    })?;
+    if amount == 0 {
+        return Err("duration must be greater than zero".to_string());
+    }
+
+    let seconds_per_unit = match unit {
+        "s" => 1,
+        "m" => 60,
+        "h" => 60 * 60,
+        _ => {
+            return Err(format!(
+                "invalid duration unit '{unit}'; use seconds (s), minutes (m), or hours (h)"
+            ));
+        }
+    };
+    amount
+        .checked_mul(seconds_per_unit)
+        .map(Duration::from_secs)
+        .ok_or_else(|| format!("duration '{value}' is too large"))
+}
+
 /// How the supervisor binary is delivered into sandbox pods.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)]
 #[serde(rename_all = "kebab-case")]
@@ -253,6 +310,8 @@ pub struct KubernetesComputeConfig {
     pub topology: SupervisorTopology,
     /// Sidecar-only settings used when `topology = "sidecar"`.
     pub sidecar: KubernetesSidecarConfig,
+    /// Operator policy for caller-requested, per-sandbox `PodDisruptionBudgets`.
+    pub disruption_protection: KubernetesDisruptionProtectionConfig,
     pub grpc_endpoint: String,
     pub ssh_socket_path: String,
     pub client_tls_secret_name: String,
@@ -346,6 +405,7 @@ impl Default for KubernetesComputeConfig {
             supervisor_sideload_method: SupervisorSideloadMethod::default(),
             topology: SupervisorTopology::default(),
             sidecar: KubernetesSidecarConfig::default(),
+            disruption_protection: KubernetesDisruptionProtectionConfig::default(),
             grpc_endpoint: String::new(),
             ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(),
             client_tls_secret_name: String::new(),
@@ -364,6 +424,12 @@ impl Default for KubernetesComputeConfig {
 }
 
 impl KubernetesComputeConfig {
+    pub fn validate_disruption_protection_config(&self) -> Result<(), String> {
+        parse_disruption_protection_duration(&self.disruption_protection.max_duration)
+            .map(|_| ())
+            .map_err(|err| format!("disruption_protection.max_duration: {err}"))
+    }
+
     /// Clamp `sa_token_ttl_secs` into the `[MIN_SA_TOKEN_TTL_SECS,
     /// MAX_SA_TOKEN_TTL_SECS]` range used by the projected-volume spec.
     /// Invalid (≤0) values fall back to the default 3600.
@@ -512,6 +578,63 @@ mod tests {
     use super::*;
     use std::collections::BTreeMap as HashMap;
 
+    #[test]
+    fn disruption_protection_is_disabled_by_default() {
+        let cfg = KubernetesComputeConfig::default();
+        assert!(!cfg.disruption_protection.enabled);
+        assert_eq!(
+            cfg.disruption_protection.max_duration,
+            DEFAULT_DISRUPTION_PROTECTION_MAX_DURATION
+        );
+    }
+
+    #[test]
+    fn parses_disruption_protection_duration_units() {
+        assert_eq!(
+            parse_disruption_protection_duration("30s").unwrap(),
+            Duration::from_secs(30)
+        );
+        assert_eq!(
+            parse_disruption_protection_duration("5m").unwrap(),
+            Duration::from_secs(300)
+        );
+        assert_eq!(
+            parse_disruption_protection_duration("4h").unwrap(),
+            Duration::from_secs(14_400)
+        );
+    }
+
+    #[test]
+    fn rejects_invalid_disruption_protection_durations() {
+        for value in ["", "0h", "4", "1d", "-1h", "1.5h"] {
+            assert!(
+                parse_disruption_protection_duration(value).is_err(),
+                "duration {value:?} should be rejected"
+            );
+        }
+    }
+
+    #[test]
+    fn validates_disruption_protection_operator_maximum() {
+        let valid: KubernetesComputeConfig = serde_json::from_value(serde_json::json!({
+            "disruption_protection": {
+                "enabled": true,
+                "max_duration": "4h"
+            }
+        }))
+        .unwrap();
+        assert!(valid.validate_disruption_protection_config().is_ok());
+
+        let invalid: KubernetesComputeConfig = serde_json::from_value(serde_json::json!({
+            "disruption_protection": {
+                "enabled": true,
+                "max_duration": "forever"
+            }
+        }))
+        .unwrap();
+        assert!(invalid.validate_disruption_protection_config().is_err());
+    }
+
     #[test]
     fn default_workspace_storage_size_is_2gi() {
         let cfg = KubernetesComputeConfig::default();