feat(core): audio file/URL constructors and representation-aware capability guards by jakelorocco · Pull Request #1601 · generative-computing/mellea

jakelorocco · 2026-09-01T16:20:41Z

Pull Request

Issue

Fixes # N/A

Description

Adds some functionality to make it easier to work with audio and adds a few guards as well. Matches functionality with the images feature.

Testing

Tests added to the respective file if code was changed
New code has 100% coverage if code was added
Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Attribution

AI coding assistants used

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

Component
Requirement
Sampling Strategy
Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.

Assisted-by: Claude Code Signed-off-by: Jake LoRocco <jake.lorocco@ibm.com>

Signed-off-by: jakelorocco <59755218+jakelorocco@users.noreply.github.com>

jakelorocco · 2026-09-01T16:57:56Z

@markstur, could you please take a look at this PR since it touches / adds some audio feature enhancements in areas you'd previously worked on?

psschwei · 2026-09-01T19:56:33Z

This is something that Claude picked up, though it seems fairly minor as I don't think we would hit this very often. Mentioning it as better safe than sorry, but could also see ignoring this:

_generate_from_intrinsic serializes context messages at openai.py:760 without a prefetch, so an AudioUrlBlock there resolves via the blocking resolve_base64() on the event loop.

The cache added here (here being L1013) makes this a hit in almost every real ordering, since any prior generation on the standard path warms it. The one case it misses is an intrinsic called on a context no generation has touched, which is the documented pattern for the intrinsic helpers (check_certainty(context, backend) over a hand-built ChatContext). Worst case is one bounded 30 s download, not wrong output.

Suggest adding this after line 748 for symmetry with the standard path:

messages: list[Message] = self.formatter.to_chat_messages(linearized_context) await prefetch_audio_urls(messages)

psschwei · 2026-09-01T20:06:55Z

-        else:
-            return None
-    else:
+    if not hasattr(c, "audio"):


Again, another minor thing that Claude found while reviewing. I think it probably makes sense to just be aware of this but not to fix as that would result in a lot of extra calls.

The fallback fires only when the attribute is absent, but message_from_template_representation (chat.py:354-364) always reads tr.audio / tr.images. So a component exposing audio = None while declaring audio=[...] on its representation passes this guard, then has the clip put on the Message by the formatter and silently dropped by HF's apply_chat_template. Same silent drop this fallback exists to prevent, different shape. test_representation_fallback_not_consulted_when_attribute_present currently pins that as intended.

No built-in can hit it: Instruction.audio and its format_for_llm both read self._audio (instruction.py:204, 229), same for Message. So this is about the extension surface, not a live bug.

Falling back whenever the attribute yields nothing, rather than only when it's missing, would make the guard agree with the payload path. That does cost the optimization the docstring calls out, since every attachment-free Instruction/Message would then pay two extra format_for_llm() calls per generation on the HF path. Fine to keep as-is and just document that a component's attribute must agree with its representation, but worth being a deliberate choice rather than an accident.

markstur

Good feature. Nice to fix the dead end AudioUrlBlock. See inline comment, however, about how vLLM can support remote URLs. Probably just FYI and no change.

I think Paul's comment should be addressed with an await prefetch.

Otherwise the doc additions, in particular, are a great addition. I had to check to see if I forgot to commit something like that (I didn't. I think I was too focused on m serve examples).

Also some little doc/comment nits inline.

markstur · 2026-09-02T04:37:36Z

+                f"Could not read audio file {os.fspath(path)[:120]!r}: expected a path "
+                f"to an audio file on disk. ({type(e).__name__}: {e}) "
+                "To load remote audio, fetch the bytes and use AudioBlock.from_bytes(); "
+                "for base64 data use AudioBlock(value, format=...)."


Instead of fetch, this guidance can now use from_url() e.g.:

To load a remote URL use AudioBlock.from_url() or AudioUrlBlock; for raw bytes use AudioBlock.from_bytes()

markstur · 2026-09-02T04:46:23Z

        self.format = format

+    def resolve_base64(self) -> str:
+        """Return the audio as raw base64, downloading it once per URL.


there is a race condition where the same URL can be downloaded multiple times in parallel and the last one just wins. Not a big concern. I think (if not fixed) a comment somewhere would be good. Maybe here?

markstur · 2026-09-02T04:49:15Z

outdates docstring. The AudioUrlBlock ValueError was removed

markstur · 2026-09-02T05:18:53Z

+
+### Remote audio
+
+No provider accepts audio by URL — OpenAI Chat Completions has no audio-by-URL content


This statement is fine for Mellea/OpenAI, but...

https://docs.vllm.ai/en/v0.6.2/getting_started/examples/openai_audio_api_client.html

So the current AudioUrlBlock was partly just nice symmetry with ImageUrlBlock even though it was a dead end, but was also a potential extension to support vLLM's non-comforming extension of OpenAI API (or maybe anticipating an OpenAI API future change?) which can do the URL download.

I'm OK with ignoring the vLLM extension, keeping this comment as-is and just saying we'll deal with that in the future (or maybe never). Probably not a priority and no need to speculate. Essentially that's why there was no issue to implement the vLLM specific audio_url.

I just want to make sure we agree that we can kick this down the road and if future us wants to add remote audio-by-URL support, we'd probably just add a check similar to is_vllm_server_with_...() to make the prefetch/cache skippable? This and that could co-exist.

planetf1

Decision drivers: none — these are non-blocking correctness and documentation comments.

Correctness (each reproduced by the reviewer): - AudioBlock.detect_format() reported ADTS AAC as mp3: AAC's syncword is 12 bits, so 0xF1 passes an 11-bit-sync-only check. Also require a non-zero MPEG layer field, which ADTS always encodes as 0b00. Verified across Layer I/II/III and MPEG 2.5. - AudioBlock.from_url() cached the download before validating its format, so a URL that transiently served a non-audio 200 stayed poisoned and no retry re-fetched. Evict the entry before raising. - AudioBlock.from_file()'s error truncated the path then interpolated the OSError, whose str() repeats the untruncated filename -- a base64 payload passed by mistake ended up whole in the exception and any log. Report the type and errno instead; a shared _truncate_for_error() bounds every echoed value. Intrinsic path: - Add the prefetch psschwei suggested so an AudioUrlBlock is resolved off-thread rather than downloading inline on the event loop. - Chasing that surfaced more: IntrinsicsRewriter validates against a strict ChatCompletion whose content must be a plain string, so any multimodal content list raised twelve pydantic errors. A plain AudioBlock already did this on main; making AudioUrlBlock resolve rather than raise extended it to URL blocks. Guard the path explicitly, mirroring LocalHFBackend._check_no_multimodal_blocks, before the prefetch so a rejected clip is never downloaded. Docs and comments left stale by the mid-PR redesigns: - from_file()'s guidance now names from_url()/AudioUrlBlock (markstur). - Drop the removed AudioUrlBlock ValueError from message_to_openai_message's Raises (markstur). - Restore the same-URL race note on the audio download cache (markstur). - 'No provider accepts audio by URL' was false -- vLLM exposes audio_url. Scope the claim to OpenAI Chat Completions and note pass-through as a possible future addition (markstur). - The from_file example printed 'mp3' directly under prose about a mislabelled WAV yielding 'wav' (planetf1). Deliberate non-change: - Document that the capability-guard fallback fires only when the attribute is absent, not when present and empty, and why -- falling back on empty would add a format_for_llm() call per attachment-free component per scan. The pinning test now records it as a trade rather than incidental behaviour (psschwei). Assisted-by: Claude Code Signed-off-by: Jake LoRocco <jake.lorocco@ibm.com>

jakelorocco · 2026-09-04T15:58:25Z

I believe I've addressed all comments.

planetf1

Approved. Two non-blocking observations are left inline for future consideration.

planetf1 · 2026-09-08T11:54:50Z

+_AUDIO_DOWNLOAD_MAX_BYTES: int = 50 * 1024 * 1024
+"""Maximum accepted size (bytes) of a downloaded audio body."""
+
+_AUDIO_CACHE_MAX_ENTRIES: int = 32


Non-blocking: this cache bounds entry count but not retained bytes. At its limits it can retain about 2.08 GiB of base64 audio. Fine for this experimental first pass, but worth revisiting as long-lived or multi-user usage grows.

planetf1 · 2026-09-08T11:54:51Z

+        if not raw:
+            raise ValueError(f"Audio at {url!r} was empty")
+    except (requests.RequestException, OSError, ValueError) as e:
+        raise ValueError(f"Failed to download audio from URL {url!r}: {e}") from e


Non-blocking: download failures include the full URL, which may contain a presigned query token. This matches the existing image URL behaviour and is reasonable for the current application-controlled API, but worth revisiting if URLs cross trust boundaries or errors go to shared telemetry.

planetf1 · 2026-09-08T11:55:03Z

+_AUDIO_DOWNLOAD_MAX_BYTES: int = 50 * 1024 * 1024
+"""Maximum accepted size (bytes) of a downloaded audio body."""
+
+_AUDIO_CACHE_MAX_ENTRIES: int = 32


Non-blocking: this cache bounds entry count but not retained bytes. At its limits it can retain about 2.08 GiB of base64 audio. Fine for this experimental first pass, but worth revisiting as long-lived or multi-user usage grows.

planetf1 · 2026-09-08T11:55:04Z

+        if not raw:
+            raise ValueError(f"Audio at {url!r} was empty")
+    except (requests.RequestException, OSError, ValueError) as e:
+        raise ValueError(f"Failed to download audio from URL {url!r}: {e}") from e


Non-blocking: download failures include the full URL, which may contain a presigned query token. This matches the existing image URL behaviour and is reasonable for the current application-controlled API, but worth revisiting if URLs cross trust boundaries or errors go to shared telemetry.

feat(core): add some user stuff to make audio easier to utilize

d9c8c71

Assisted-by: Claude Code Signed-off-by: Jake LoRocco <jake.lorocco@ibm.com>

github-actions Bot added the enhancement New feature or request label Sep 1, 2026

Merge branch 'main' into feat/audio-input

eb21284

Signed-off-by: jakelorocco <59755218+jakelorocco@users.noreply.github.com>

jakelorocco marked this pull request as ready for review September 1, 2026 16:57

jakelorocco requested a review from a team as a code owner September 1, 2026 16:57

jakelorocco requested review from ajbozarth, markstur and planetf1 September 1, 2026 16:57

psschwei reviewed Sep 1, 2026

View reviewed changes

markstur reviewed Sep 2, 2026

View reviewed changes

planetf1 reviewed Sep 3, 2026

View reviewed changes

Comment thread mellea/core/base.py

Comment thread mellea/core/base.py Outdated

Comment thread mellea/core/base.py Outdated

Comment thread docs/docs/how-to/use-speech-and-audio.md Outdated

psschwei assigned nrfulton Sep 4, 2026

planetf1 approved these changes Sep 8, 2026

View reviewed changes

planetf1 reviewed Sep 8, 2026

View reviewed changes


		### Remote audio

		No provider accepts audio by URL — OpenAI Chat Completions has no audio-by-URL content

Uh oh!

Conversation

jakelorocco commented Sep 1, 2026

Pull Request

Issue

Description

Testing

Attribution

Adding a new component, requirement, sampling strategy, or tool?

Uh oh!

jakelorocco commented Sep 1, 2026

Uh oh!

psschwei Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markstur left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

planetf1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jakelorocco commented Sep 4, 2026

Uh oh!

planetf1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

psschwei Sep 1, 2026 •

edited

Loading