fix(inference): prepend publisher prefix for Vertex non-Anthropic models by politerealism · Pull Request #2735 · NVIDIA/OpenShell

politerealism · 2026-08-13T15:41:27Z

Summary

Fixes the deadlock in google-vertex-ai provider type cannot route to Gemini models — publisher-prefix requirement conflicts with model ID validation #2351 where gemini-2.5-flash passes validation but Vertex rejects it with HTTP 400 "Malformed publisher model", and google/gemini-2.5-flash would be accepted by Vertex but is rejected by CLI validation (slash guard)
In resolve_vertex_ai_route, computes a body_model_id for non-Anthropic routes by prepending the publisher from infer_vertex_publisher() (e.g. google for Gemini, meta for Llama) or the explicit VERTEX_AI_PUBLISHER config value — the bare model_id still goes through the path-traversal validator unchanged
Applies to both the project/region path and the base-URL-override path; Anthropic rawPredict routes are unaffected (model encoded in URL, not body)
Adds a backend.rs integration test with a mock Vertex server that accepts only google/gemini-2.5-flash and explicitly expects zero hits on the bare model name

Related Issue

Changes

File	Change
`crates/openshell-server/src/inference.rs`	Compute `body_model_id` with publisher prefix in non-Anthropic branch; update existing tests; add 4 new unit tests
`crates/openshell-router/src/backend.rs`	Add integration test with mock Vertex endpoint verifying publisher-prefixed body model field

Testing

cargo check -p openshell-server — clean
cargo clippy -p openshell-server -p openshell-router — clean
cargo test -p openshell-router — 18 passed
Integration test: mock Vertex server accepts prefixed form, expect(0) on bare model ensures pre-fix path is never hit

Checklist

Conventional Commits format
DCO Signed-off-by present
No unrelated changes
Existing tests updated to assert new behaviour; no tests deleted

copy-pr-bot · 2026-08-13T15:41:32Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-08-13T15:41:40Z

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

politerealism · 2026-08-13T20:07:13Z

recheck

politerealism · 2026-08-13T20:41:23Z

recheck

johntmyers · 2026-08-13T23:18:21Z

gator-agent

Blocked

Gator is blocked because the current-head DCO check is failing: commit ae9c2211b10f591283d6a1e1f800c54b2f2cc8fb is authored as Quinn Burdine with sburdine@redhat.com, but GitHub does not associate that commit author identity with a GitHub account.

Head SHA: ae9c2211b10f591283d6a1e1f800c54b2f2cc8fb
Gator payload: 4

Next action: @politerealism, add sburdine@redhat.com to the GitHub account that owns this commit (or replace the commit with one using an email already associated with that account), then comment recheck so DCO Assistant reruns.

johntmyers

gator-agent

PR Review Status

Validation: Project-valid localized fix for confirmed Vertex AI routing bug #2351.
Head SHA: 54020761b65bb32dc423828f8fb7d6ac8f5ae131
Base SHA: c5498239e66581d5842b50773eff07ace65726f1
Merge base SHA: 0f8fad23c4712afc1d4a7b07a06d635b030e9521
Patch ID: d7629ffbecc6e86cc445e539e5dccd096341a601
Gator payload: 4
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no

Blocking findings:

GATOR-54020761-01: the publisher-prefixed model is lost at the inference-bundle boundary, so live sandbox requests still use the bare model.

Carried findings:

None

Non-blocking suggestions:

None

Docs: No docs update is needed for this fix because it restores the already documented Vertex non-Anthropic routing path without changing configuration or user workflow.

Next state: gator:in-review

johntmyers · 2026-08-14T00:10:26Z

            route_name,
            base_url,
-            model_id,
+            &body_model_id,


gator-agent

Warning — GATOR-54020761-01

Invariant: The publisher-prefixed Vertex model identifier must survive route resolution into the inference bundle consumed by the supervisor router.

Prerequisite: An operator configures a supported non-Anthropic google-vertex-ai route whose publisher is inferred or supplied with VERTEX_AI_PUBLISHER.

Entry point → sink: openshell inference set --provider <vertex-provider> --model gemini-2.5-flash, followed by an OpenAI-compatible request through inference.local → the supervisor converts GetInferenceBundleResponse.model_id into ResolvedRoute.model, which the router writes into the outbound Vertex request body.

Base → head: Base carried the bare model through resolution and runtime. Head creates the prefixed value here, so endpoint verification can use it, but resolve_route_by_name_with_credentials still exports model_id: config.model_id.clone(); the supervisor therefore reconstructs the live route with the bare model.

Impact: Live Gemini, Llama, Mistral, AI21, DeepSeek, or explicitly published Vertex requests still fail with a malformed publisher model even when endpoint verification succeeds.

Reproducer: Store a Vertex provider and inference.local route for gemini-2.5-flash, resolve the inference bundle, and assert its route model_id is google/gemini-2.5-flash. At this head it remains gemini-2.5-flash; converting the bundle and preparing a request emits the bare model.

PR ownership: This PR introduces body_model_id specifically to repair runtime body shaping, but the value is not propagated across the production server-to-supervisor bundle boundary. The added router test manually constructs a prefixed route and bypasses that boundary.

Requested change: Populate the resolved bundle's model_id from resolved.route.model while retaining the configured bare ID only in persisted/user-facing configuration, and add a bundle-path test proving the supervisor receives the prefixed model. This applies to both system/user routes and both Vertex endpoint construction branches.

johntmyers

gator-agent

PR Review Status

Thanks @politerealism. I reviewed the update that propagates the resolved publisher-prefixed model through the inference bundle and adds the bundle-path regression test.

Validation: Project-valid localized fix for confirmed Vertex AI routing bug #2351.
Head SHA: aff922eb165e91563dfbc0d832c01dc4b0b58d2f
Base SHA: f12f3ef8d565b4b431de4f3caad996022f62e42b
Merge base SHA: f12f3ef8d565b4b431de4f3caad996022f62e42b
Patch ID: 72604ee869b3df018f6d17d95f3875b1f731a1db
Gator payload: 4
Review mode: follow_up
Previous reviewed SHA: 54020761b65bb32dc423828f8fb7d6ac8f5ae131
Review budget exhausted: no
Maintainer decision required: no

Blocking findings:

No blocking findings remain.

Carried findings:

GATOR-54020761-01: resolved. The bundle now uses resolved.route.model, and the new regression test verifies google/gemini-2.5-flash survives the bundle boundary.

Docs: No docs update is needed because this restores the existing Vertex non-Anthropic routing path without changing configuration or user workflow.

Next state: gator:in-review pending required E2E workflow dispatch.

github-actions · 2026-08-14T20:03:49Z

Label test:e2e applied, but pull-request/2735 does not exist yet. A maintainer needs to comment /ok to test aff922eb165e91563dfbc0d832c01dc4b0b58d2f to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

johntmyers · 2026-08-14T20:19:41Z

/ok to test aff922e

johntmyers · 2026-08-14T21:14:06Z

/ok to test 28868f4

johntmyers

gator-agent

PR Review Status

Thanks @politerealism. I reviewed the force-pushed current head: the two previously reviewed functional commits are preserved, and the only new author-only delta applies rustfmt to the bundle regression test.

Validation: Project-valid localized fix for confirmed Vertex AI routing bug #2351.
Head SHA: 28868f4d23df0b416dc5fdd37ea3f318ebc0a730
Base SHA: bdabb54cb361a59d4eba957a8cdfd85ae5858289
Merge base SHA: bdabb54cb361a59d4eba957a8cdfd85ae5858289
Patch ID: 37a65eb68483a7c4645bd21a47e3946c4eaf6108
Gator payload: 4
Review mode: follow_up
Previous reviewed SHA: aff922eb165e91563dfbc0d832c01dc4b0b58d2f
Review budget exhausted: no
Maintainer decision required: no

Blocking findings:

No blocking findings remain.

Carried findings:

GATOR-54020761-01: remains resolved. The current patch still propagates resolved.route.model through the inference bundle, and the regression test continues to verify the publisher-prefixed model at that boundary.

Docs: No docs update is needed because this restores the existing Vertex non-Anthropic routing path without changing configuration or user workflow.

Required CI, including test:e2e, is now queued for this exact head after the authorized /ok to test mirror dispatch.

Next state: gator:watch-pipeline

Vertex AI's OpenAI-compatible endpoint requires the request body's model field to carry a publisher prefix (e.g. google/gemini-2.5-flash), but validate_vertex_model_id rejects slash as a path-traversal guard. This created a deadlock: bare model IDs pass validation but are rejected by Vertex with HTTP 400 "Malformed publisher model"; prefixed IDs are rejected at configuration time. Fix: in resolve_vertex_ai_route, compute body_model_id for non-Anthropic routes by prepending the publisher from infer_vertex_publisher() or the explicit VERTEX_AI_PUBLISHER config value. The bare model_id still goes through the path-traversal validator unchanged. Anthropic rawPredict routes encode the model in the URL path, not the body, and are unaffected. Both the project/region path and the base-URL-override path apply the prefix. For unrecognised models with no explicit publisher the bare ID is forwarded unchanged; Vertex's 400 is the correct observable signal in that case. Add an integration test in openshell-router that spins up a mock Vertex endpoint accepting only the publisher-prefixed form and rejecting the bare model name, verifying the body rewrite produces the required format. Closes NVIDIA#2351 Signed-off-by: politerealism <burdcat17@gmail.com>

…ce bundle resolve_route_by_name_with_credentials built the ResolvedRoute with config.model_id (the bare stored value) rather than resolved.route.model (the publisher-prefixed value computed by resolve_vertex_ai_route). As a result, the bundle delivered to sandboxes carried e.g. "gemini-2.5-flash" instead of "google/gemini-2.5-flash", so live sandbox requests still hit Vertex AI with the bare model name and received HTTP 400 "Malformed publisher model". Fix: use resolved.route.model in the bundle construction so the publisher prefix survives the bundle boundary and the router sends the correct body to Vertex AI. Update the existing gemini bundle test to assert the prefixed model_id and add a dedicated regression test that verifies the bundle carries the publisher prefix for non-Anthropic Vertex routes. Signed-off-by: politerealism <burdcat17@gmail.com>

Signed-off-by: politerealism <burdcat17@gmail.com>

- Invert if !is_anthropic to satisfy clippy::if_not_else - Replace match on Option with map_or_else to satisfy clippy::option_if_let_else Signed-off-by: politerealism <burdcat17@gmail.com>

johntmyers · 2026-08-14T22:41:13Z

/ok to test 591198e

johntmyers

gator-agent

PR Review Status

Thanks @politerealism. I reviewed the current-head Clippy cleanup after comparing the author patch to the previously reviewed head. The three earlier commits are patch-equivalent; the only new author delta inverts the Anthropic branch and replaces the equivalent Option match with map_or_else.

Validation: Project-valid localized fix for confirmed Vertex AI routing bug #2351.
Head SHA: 591198eff8a168686bf48b99d3d21958c1d20699
Base SHA: 44bf0df485f15c2992b04c1d597e6967b4b37bcd
Merge base SHA: 44bf0df485f15c2992b04c1d597e6967b4b37bcd
Patch ID: 0ebdaa85c08b23fca41af817be38eb64b3ce00d6
Gator payload: 4
Review mode: critical_only
Previous reviewed SHA: 28868f4d23df0b416dc5fdd37ea3f318ebc0a730
Review budget exhausted: yes
Maintainer decision required: no — no unresolved obligation, qualifying scope growth, or proposed Warning remains.

Blocking findings:

No blocking findings remain. The critical-only delta review found no newly introduced Critical defect.

Carried findings:

GATOR-54020761-01: remains resolved. The patch still propagates resolved.route.model through the inference bundle, and the regression test continues to cover the publisher-prefixed model at that boundary.

Docs: No docs update is needed because this restores the existing Vertex non-Anthropic routing path without changing configuration or user workflow.

Required current-head CI is queued after the authorized mirror dispatch, including test:e2e.

Next state: gator:watch-pipeline

politerealism requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners August 13, 2026 15:41

politerealism force-pushed the fix/vertex-gemini-publisher-prefix branch from 9fe55d5 to 1f14173 Compare August 13, 2026 16:40

politerealism force-pushed the fix/vertex-gemini-publisher-prefix branch from 1f14173 to ae9c221 Compare August 13, 2026 20:11

johntmyers added the gator:blocked Gator is blocked by process or repository gates label Aug 13, 2026

politerealism force-pushed the fix/vertex-gemini-publisher-prefix branch from ae9c221 to 5402076 Compare August 13, 2026 23:54

johntmyers reviewed Aug 14, 2026

View reviewed changes

johntmyers added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:blocked Gator is blocked by process or repository gates labels Aug 14, 2026

politerealism force-pushed the fix/vertex-gemini-publisher-prefix branch from 5402076 to aff922e Compare August 14, 2026 19:48

johntmyers reviewed Aug 14, 2026

View reviewed changes

johntmyers added the test:e2e Requires end-to-end coverage label Aug 14, 2026

johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Aug 14, 2026

politerealism force-pushed the fix/vertex-gemini-publisher-prefix branch from aff922e to 28868f4 Compare August 14, 2026 20:59

johntmyers reviewed Aug 14, 2026

View reviewed changes

johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status and removed gator:in-review Gator is reviewing or awaiting PR review feedback labels Aug 14, 2026

johntmyers added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Aug 14, 2026

politerealism added 4 commits August 14, 2026 18:09

style(inference): apply rustfmt to new regression test

eab2fe9

Signed-off-by: politerealism <burdcat17@gmail.com>

fix(inference): address clippy lints in build_vertex_route

591198e

- Invert if !is_anthropic to satisfy clippy::if_not_else - Replace match on Option with map_or_else to satisfy clippy::option_if_let_else Signed-off-by: politerealism <burdcat17@gmail.com>

politerealism force-pushed the fix/vertex-gemini-publisher-prefix branch from 28868f4 to 591198e Compare August 14, 2026 22:09

politerealism requested a review from sjenning as a code owner August 14, 2026 22:09

johntmyers reviewed Aug 14, 2026

View reviewed changes

Uh oh!

Conversation

politerealism commented Aug 13, 2026

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Aug 13, 2026

Uh oh!

github-actions Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

politerealism commented Aug 13, 2026

Uh oh!

politerealism commented Aug 13, 2026

Uh oh!

johntmyers commented Aug 13, 2026

Blocked

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

johntmyers Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

johntmyers commented Aug 14, 2026

Uh oh!

johntmyers commented Aug 14, 2026

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

johntmyers commented Aug 14, 2026

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Aug 13, 2026 •

edited

Loading