fix(backends): reject empty user content before sending chat request - #1632
Draft
simonyang08 wants to merge 1 commit into
Draft
Conversation
SimpleContext.view_for_generation() returns [] by design (stateless context), so a caller who chains .add(...) and then passes an empty or whitespace-only action to generate_from_context used to ship an empty user-role conversation to the model. Some chat models — Granite 4.2 in particular, see issue generative-computing#1587 — spin on an empty prompt and burn tokens silently, which is hard to diagnose in CI. Raise ValueError in three concrete chat assembly paths: * OpenAIBackend._generate_from_chat_context_standard * LiteLLMBackend._generate_from_chat_context_standard * OllamaModelBackend.generate_from_chat_context The check ignores whitespace-only strings but still accepts user messages that carry images, audio, or documents (vision / RAG paths must continue to work). Each guard raises a ValueError before any HTTP or SDK call is issued. WatsonxAIBackend (mellea/backends/watsonx.py) and LocalHFBackend (mellea/backends/huggingface.py via mellea/backends/utils.py:to_chat) also assemble user-role conversations and would benefit from the same guard, but they are intentionally left untouched to keep this change focused on the backends named in the issue. Happy to mirror the same guard block in those two backends in a follow-up if preferred. SimpleContext.add's docstring is updated to make the stateless semantics explicit (recorded turns are never forwarded; the action argument is the only thing that reaches the model) and to call out which backends now enforce the invariant. Closes generative-computing#1597 Assisted-by: ZCode (GLM) Signed-off-by: simonyang08 <ppt5928@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1597
Root cause
SimpleContext.view_for_generation()returns[]by design — it is a stateless context. But.add()still exists and returns a chainable context, which reads like it records the turn for the next call. A caller who.add()s a user message and then passes an empty or whitespace-onlyactiontogenerate_from_contextends up sending the model a conversation with no user-role content at all, and nothing errors. This is exactly what happened in #1587's live Ollama telemetry tests, where Granite 4.2 rambled for 2000+ tokens on the resulting empty prompt and stalled CI.Fix
Add a shared invariant guard in the chat-message assembly path of the three backends that build user-role conversations for chat APIs —
OpenAIBackend._generate_from_chat_context_standard,LiteLLMBackend._generate_from_chat_context_standard, andOllamaModelBackend.generate_from_chat_context:messageslist has no user-role message with non-whitespace content, raiseValueErrorbefore any HTTP/SDK call is issued;images,audio, or documents (_docs) still pass, so vision/RAG paths keep working.ValueErrormatches the existing repo convention for "call cannot proceed" (cf.context_lengths.py,intrinsic/_util.py).SimpleContext.add/view_for_generationdocstrings now state explicitly that recorded turns are never forwarded, so the.add()-then-empty-action trap is documented at the API surface too.Note on scope:
WatsonxAIBackendandLocalHFBackendalso assemble user-role conversations and would benefit from the same guard, but are intentionally left untouched to keep this change focused on the backends named in the issue. Happy to mirror the guard in a follow-up if preferred.Verification
New regression suite
test/backends/test_simple_context_guard.py(mocked providers, no network):DID NOT RAISE ValueError), green after the fix;imagespasses the guard (multimodal path);guardian.pycall pattern (non-empty context + empty assistant action) is not affected.Focused backend/context suites: 170 passed.
test/stdlib/regression run shows no new failures relative to unpatched main.DCO:
Signed-off-by: simonyang08 <ppt5928@gmail.com>. AI-assisted contribution per CONTRIBUTING.md (Assisted-by: ZCode (GLM)trailer on the commit).