Skip to content

feat(kubernetes): support corporate upstream proxy - #2633

Merged
johntmyers merged 3 commits into
NVIDIA:mainfrom
loveRhythm1990:2624-kubernetes-corporate-proxy/lr
Aug 14, 2026
Merged

feat(kubernetes): support corporate upstream proxy#2633
johntmyers merged 3 commits into
NVIDIA:mainfrom
loveRhythm1990:2624-kubernetes-corporate-proxy/lr

Conversation

@loveRhythm1990

@loveRhythm1990 loveRhythm1990 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Add Kubernetes support for the existing supervisor corporate HTTP forward-proxy path. This lets sandboxes in proxy-only clusters reach policy-approved TLS destinations through an operator-managed proxy without allowing workloads to choose, bypass, or observe the upstream route.

The Kubernetes driver previously had no configuration or Secret-projection path for the shared supervisor feature that Podman already uses. This PR adds that driver-level plumbing while preserving direct egress as the default when no proxy is configured.

Related Issue

Closes #2624

Changes

  • Add Kubernetes driver configuration for https_proxy, no_proxy, proxy credential Secret name/key, insecure Basic-auth acknowledgement, and CONNECT-by-hostname mode. The URL remains intentionally limited to http:// forward proxies; HTTPS-to-proxy support is out of scope.
  • Expose the configuration through gateway CLI/environment options and Helm upstreamProxy values, while keeping it deployment-owned rather than allowing per-sandbox driver_config overrides.
  • Reuse the shared URL validation and enforce coherent settings at gateway startup: invalid URLs, unpaired Secret fields, invalid Secret names/keys, and unsafe credential settings fail closed.
  • Project credentials as a read-only Kubernetes Secret volume, never as an environment value, annotation, or command-line value. The gateway validates the reference syntax; kubelet resolves Secret existence/key availability when the sandbox Pod starts; the supervisor validates the credential content and rejects malformed files.
  • Require topology = \"sidecar\" whenever proxy credentials are configured. Combined topology shares a container filesystem with the workload, and workspace fsGroup can make an otherwise root-only Secret mount group-readable; rejecting this combination avoids a credential disclosure path. Credential-free proxy configuration can still use Combined topology.
  • Pass upstream arguments only to the network-supervising container. In sidecar mode this is the network sidecar; network-init never receives proxy configuration and the agent container never receives the credential mount.
  • Update the gateway reference, Kubernetes setup guide, compute-driver reference, architecture overview, Helm README, values, and rendering tests.

Testing

  • mise run pre-commit passes
  • Kubernetes driver unit tests and gateway TOML/Helm rendering tests updated
  • Kubernetes mock-proxy e2e: authenticated CONNECT reaches a TLS upstream and policy-denied traffic never reaches the proxy
  • Kubernetes mock-proxy e2e: NO_PROXY bypasses only the corporate proxy; policy-denied traffic remains denied
  • Kubernetes e2e: missing credential Secret fails sandbox creation closed
  • Kubernetes e2e: malformed credential content fails sandbox creation closed
  • mise run test has one unrelated environment-dependent DNS failure: test_forward_public_ip_allowed_without_allowed_ips resolved dns.google to 198.18.1.240, which the SSRF guard correctly rejects. All Kubernetes proxy-specific checks above passed.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@copy-pr-bot

copy-pr-bot Bot commented Aug 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

@loveRhythm1990
loveRhythm1990 marked this pull request as ready for review August 6, 2026 13:48
@loveRhythm1990
loveRhythm1990 force-pushed the 2624-kubernetes-corporate-proxy/lr branch 2 times, most recently from 26493d1 to 78250e4 Compare August 6, 2026 13:52

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

Validation: Project-valid as a focused Kubernetes-driver extension of the corporate upstream-proxy support merged in #2245, with the Kubernetes-specific operator path and motivation documented in #2624.
Head SHA: 78250e45a3ade0622f167be52f1499e190ecc917
Base SHA: 0c7e59a95355cabc15ccaddb86fcbe6a1d30eaaa
Merge base SHA: d2c44b0e5393e3746eae5783aa99ac31be08daae
Patch ID: 5db52dea1e5bdc062384aa8edc68b640bfa32b9d
Gator payload: 4
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no

Blocking findings:

  • GATOR-78250e45-01: Kubernetes Secret keys longer than 253 bytes pass gateway validation even though Kubernetes cannot create them, shifting an invalid deployment setting into repeated sandbox Pod-provisioning failures.

Carried findings:

  • None

Non-blocking suggestions:

  • Consider adding corporate-proxy checks to debug-openshell-cluster for rendered configuration, Secret-volume events, supervisor arguments/mounts, and proxy reachability.

Docs: Fern documentation covers the new Kubernetes setup and gateway/compute-driver configuration; existing navigation already exposes those pages.

Next state: gator:in-review

Comment thread crates/openshell-driver-kubernetes/src/config.rs
@johntmyers johntmyers added the gator:in-review Gator is reviewing or awaiting PR review feedback label Aug 13, 2026
@johntmyers

Copy link
Copy Markdown
Collaborator

Thanks @loveRhythm1990 - started our review agent. I'm also soliciting some reviews from other maintainers.

@russellb

Copy link
Copy Markdown
Contributor

I took a pass on this and wasn't able to identify any issues beyond the one already highlighted by John's agent. Nice work

Signed-off-by: loveRhythm1990 <qiuweimin@126.com>
…t create

Gateway validation accepted proxy_auth_secret_key values that Kubernetes
rejects when creating the Secret (keys longer than 253 bytes, or the
reserved "."/".." names), turning an invalid deployment setting into
repeated sandbox Pod-provisioning failures instead of a startup error.
Reject them in validate_upstream_proxy_config so they fail closed at
gateway startup.

Signed-off-by: loveRhythm1990 <qiuweimin@126.com>
…luster

Add a Kubernetes corporate upstream proxy troubleshooting section covering
rendered [openshell.drivers.kubernetes] configuration, credential Secret
volume events, supervisor arguments and mounts confined to the network-
supervising container, and proxy reachability.

Signed-off-by: loveRhythm1990 <qiuweimin@126.com>
@loveRhythm1990
loveRhythm1990 force-pushed the 2624-kubernetes-corporate-proxy/lr branch from 78250e4 to f727a8d Compare August 14, 2026 04:51
@loveRhythm1990

Copy link
Copy Markdown
Contributor Author

recheck

@johntmyers johntmyers added the test:e2e Requires end-to-end coverage label Aug 14, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/2633 does not exist yet. A maintainer needs to comment /ok to test f727a8d7128ff46cd8222482fdc1cf9893887bb7 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

Thanks @loveRhythm1990. I checked your update that rejects Kubernetes Secret keys over 253 bytes and the reserved ./.. names, including the 253-byte accepted and 254-byte rejected boundary tests. That resolves GATOR-78250e45-01. I also checked the author-only troubleshooting-skill update; it addresses the earlier non-blocking suggestion.

Validation: Project-valid as a focused Kubernetes-driver extension of the corporate upstream-proxy support, with the Kubernetes-specific operator path and motivation documented in accepted issue #2624.
Head SHA: f727a8d7128ff46cd8222482fdc1cf9893887bb7
Base SHA: c4b500a7de64d0b66e3ee8098f58d14299092162
Merge base SHA: c4b500a7de64d0b66e3ee8098f58d14299092162
Patch ID: d090894f5d8ca10c405cc12c83dc9c82038f991a
Gator payload: 4
Review mode: follow_up
Previous reviewed SHA: 78250e45a3ade0622f167be52f1499e190ecc917
Review budget exhausted: no
Maintainer decision required: no

Blocking findings:

  • No blocking findings remain.

Carried findings:

  • GATOR-78250e45-01: resolved by the latest author delta.

Docs: Fern documentation covers the user-visible Kubernetes proxy configuration, and the debug-cluster skill now covers rendered configuration, Secret-volume events, supervisor arguments and mounts, and proxy reachability.

Next state: gator:in-review pending confirmation that the newly requested current-head E2E workflow has been queued, started, or completed.

@johntmyers

Copy link
Copy Markdown
Collaborator

/ok to test f727a8d

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Aug 14, 2026
@johntmyers
johntmyers added this pull request to the merge queue Aug 14, 2026
@johntmyers johntmyers added gator:merge-ready and removed gator:approval-needed Gator completed review; maintainer approval needed labels Aug 14, 2026
Merged via the queue into NVIDIA:main with commit d0c6dc3 Aug 14, 2026
67 of 71 checks passed
@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Head SHA: f727a8d7128ff46cd8222482fdc1cf9893887bb7
Gator payload: 4

Final status: The PR reached gator:merge-ready after the Kubernetes corporate-upstream-proxy review finding was resolved, current-head E2E completed, required checks passed, and maintainer approval was recorded.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(kubernetes): support corporate HTTP forward proxy egress for sandboxes

3 participants