feat(web-ui): automatic deinterlacing with bwdif and codec-metadata activation#610
Open
stackia wants to merge 10 commits into
Open
feat(web-ui): automatic deinterlacing with bwdif and codec-metadata activation#610stackia wants to merge 10 commits into
stackia wants to merge 10 commits into
Conversation
…ction Render-side deinterlacing on top of the existing MSE pipeline: a requestVideoFrameCallback + WebGL2 overlay draws deinterlaced frames to a per-slot canvas while the video element keeps driving playback and audio. - Heuristic detection (no metadata): periodic luma sampling with a classic comb metric, M-of-N rolling window, sticky verdict, reset on stream switch - Gated to 1080-class content (height 1080/1088, width <= 1920) - Pluggable algorithm registry; initial bob renders at field rate (25i->50p) with separate chroma low-pass to remove field-interleaved color combing - Settings toggle (auto/off) persisted to player storage - devlab: interlaced 1080i scan channel plus 1080p/2160p gating controls
GLSL port of FFmpeg's bwdif filter_line: per-pixel motion adaptation keeps both fields in still areas (full vertical resolution, no bob shimmer on static detail like logos/subtitles) and reconstructs moving areas with the edge-preserving spatio-temporal filter, clamped to the temporal neighborhood. Runs one frame behind the video (3-frame texture ring) at field rate for 50p motion. Chroma keeps the separate vertical low-pass since field-interleaved chroma combing from progressive 4:2:0 upsampling also sits on kept lines.
BFF sources played with a TFF assumption produce two-forward-one-back motion judder at field rate. Add a field-order vote to the detector: capture two consecutive frames via requestVideoFrameCallback and compare TFF vs BFF hypotheses by their temporal-midpoint interpolation error (the wrong order averages fields two field-times apart, so with motion its error is larger). Majority vote with abstention on still/ambiguous pairs; activation starts immediately with the TFF default and re-emits if BFF wins. Also lower the combed-frame ratio to 0.01 (in-browser measurements: interlaced 0.018-0.022 vs progressive 0.004-0.005; the old 0.02 threshold made detection flaky on frames with moderate motion) and add a 1080i BFF devlab channel. Verified in Chrome against devlab: tff=4/0 on the TFF channel, bff=4/0 on the BFF channel, no false positive on 1080p.
Review against libavfilter/bwdifdsp.c + vf_bwdif.c found three gaps: - Frame-boundary rows: FFmpeg dispatches y<4 / y+5>h to FILTER_EDGE (plain (c+e)/2 with the spatial check only where its +-2 taps fit) instead of running the +-3/+-4-tap filters into duplicated edge rows; the shader now mirrors that row dispatch instead of relying on texture clamping. - Chroma was an unconditional vertical low-pass, softening static color detail that FFmpeg's per-plane weave path preserves. Now motion-adaptive: static pixels pass original chroma through, moving pixels ramp toward the [1,2,2,2,1]/8 low-pass (true per-plane bwdif is impossible on RGB frames -- the progressive 4:2:0 upsample bakes field-mixed chroma into all rows). - Documented the weave-threshold float mapping (FFmpeg's integer !diff test == 0.5/255) and the intentional RGB-vs-YUV deviations. Verified in Chrome on the 1080i TFF channel: moving comb 0.0026, paused still 0.0000, static chroma edges preserved.
…e paused frame - Parse H.264 frame_mbs_only_flag and H.265 field_seq/interlaced_source flags, plumb them demuxer -> remuxer -> worker -> player 'video-info' event -> deinterlace pipeline. Eligible interlaced streams now start deinterlaced immediately instead of waiting for the combing heuristic warm-up; the heuristic remains for streams with progressive metadata but combed content. - Prime the renderer with the current video frame on start(), so re-enabling deinterlacing while paused replaces the stale canvas frame from a previous run. Until the texture ring holds real history the shader falls back to spatial-only interpolation (temporal terms would compare duplicated frames).
The React player no longer manages deinterlace pipelines, video-info hinting, or verdict resets. It passes an overlay canvas and mode via PlayerConfig (deinterlaceCanvas / deinterlaceMode), toggles mode at runtime with Player.setDeinterlaceMode(), and reacts to the new deinterlace-active-change event for visibility styling. The core resets the interlace verdict on loadSegments/stop and hints the detector from codec metadata internally. The canvas is stripped from the config posted to the transmux worker (not structured-cloneable). Also raise COMB_PIXEL_THRESHOLD 121 -> 400 (~±20 luma levels): sharp progressive 1080p detail scored up to 0.011 at the old threshold and false-positived the detector; measured in-browser the new threshold keeps progressive <=0.005 while weaved interlaced stays >=0.011.
…rlace detection Weaved TFF content encoded and flagged as progressive (no fieldorder filter, no interlaced encoder flags), so the codec-metadata hint stays silent and only the heuristic comb detector can activate deinterlacing.
…pipeline to boolean
Contributor
Documentation previewThe documentation preview has been deployed for this pull request. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fa706bc0db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+201
to
+203
| this.interlaced = true; | ||
| Log.i(TAG, `Interlaced per codec metadata hint (${width}x${height})`); | ||
| this.onVerdict({ interlaced: true, algorithm: "bwdif", fieldOrder: this.fieldOrder }); |
There was a problem hiding this comment.
For ≤1080 streams where the SPS/VUI flags only say pictures may be interlaced (for example H.264 frame_mbs_only_flag == 0 while the actual pictures are progressive), this turns the hint into a sticky interlaced verdict without any comb-score confirmation. Those progressive sources will have the raw video hidden and every frame processed by bwdif for the whole channel, causing avoidable blur/judder instead of falling back to the heuristic path.
Useful? React with 👍 / 👎.
Comment on lines
+79
to
+82
| if (this.video.readyState >= HTMLMediaElement.HAVE_CURRENT_DATA) { | ||
| this.renderFrame(this.fieldOrder === "tff" ? 0 : 1, false); | ||
| } | ||
| this.scheduleFrame(); |
There was a problem hiding this comment.
When the codec hint arrives, video-info is emitted before the init segment/media data, so on normal tune-in video.readyState is still below HAVE_CURRENT_DATA. This branch is skipped but start() still returns true; setActive then emits active and VideoPlayer applies opacity-0 to the video while showing a canvas that has never been drawn, producing a black frame (or lasting until playback advances) for metadata-hinted streams.
Useful? React with 👍 / 👎.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an automatic, render-side deinterlacing pipeline to the Web UI MPEG-TS player. It detects interlaced streams via codec metadata hints and/or a heuristic combing detector, then renders deinterlaced frames to a WebGL2 overlay canvas while leaving the MSE pipeline and audio timing driven by the original <video> element.
Changes:
- Introduces a new WebGL2 deinterlacing pipeline (detector + renderer + bwdif GLSL algorithm) and plumbs codec-level “may be interlaced” metadata from demux → remux → worker → player events.
- Adds runtime user control (auto/on vs off) persisted in player storage and surfaced in the player settings UI.
- Extends devlab with multicast “scan” channels to exercise interlace detection, field order, and resolution gating.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| web-ui/src/pages/player.tsx | Wires persisted “deinterlace” setting into the player page and settings dropdown props. |
| web-ui/src/mpegts/worker/transmux-worker.ts | Emits a new video-info worker event derived from init segment metadata. |
| web-ui/src/mpegts/worker/messages.ts | Adds video-info worker event type (width/height + interlace hint). |
| web-ui/src/mpegts/types.ts | Defines VideoTrackInfo, new player events, and setDeinterlace() API. |
| web-ui/src/mpegts/remux/mp4-remuxer.ts | Attaches videoInfo (dimensions + interlace hint) to video init segments. |
| web-ui/src/mpegts/player/mpegts-player.ts | Forwards video-info messages to the player impl without breaking init batching. |
| web-ui/src/mpegts/index.ts | Creates/owns the deinterlace pipeline, forwards hints, exposes events and setDeinterlace(). |
| web-ui/src/mpegts/demux/ts-demuxer.ts | Sets mayBeInterlaced in track metadata from H.264/H.265 parsed flags. |
| web-ui/src/mpegts/demux/sps-parser.ts | Exposes H.264 frame_mbs_only_flag for interlace-capability hinting. |
| web-ui/src/mpegts/demux/h265-parser.ts | Exposes an HEVC “interlaced-capable” hint from VUI/PTL flags. |
| web-ui/src/mpegts/deinterlace/renderer.ts | New: WebGL2 renderer driven by requestVideoFrameCallback with field-rate output. |
| web-ui/src/mpegts/deinterlace/index.ts | New: Pipeline wiring detector verdicts to renderer activation and events. |
| web-ui/src/mpegts/deinterlace/detector.ts | New: Comb-metric heuristic detector + field-order voting logic. |
| web-ui/src/mpegts/deinterlace/algorithms/types.ts | New: Algorithm registry + interface for pluggable GPU deinterlacers. |
| web-ui/src/mpegts/deinterlace/algorithms/gl-utils.ts | New: Shared shader/program helpers and fullscreen triangle vertex shader. |
| web-ui/src/mpegts/deinterlace/algorithms/bwdif.ts | New: GLSL bwdif implementation (motion-adaptive deinterlacing). |
| web-ui/src/mpegts/config.ts | Adds deinterlaceCanvas and deinterlace config knobs with defaults. |
| web-ui/src/lib/player-storage.ts | Persists the deinterlacing setting in local storage. |
| web-ui/src/i18n/player.ts | Adds i18n strings for the deinterlacing toggle. |
| web-ui/src/components/player/video-player.tsx | Adds overlay canvases per slot, hooks player events, toggles runtime deinterlacing. |
| web-ui/src/components/player/settings-dropdown.tsx | Adds the deinterlacing toggle UI control. |
| tools/devlab/README.md | Documents new “interlace-scan” multicast channels for deinterlacing testing. |
| tools/devlab/devlab.py | Adds scan-channel generation (1080i TFF/BFF, combed progressive, 2160p gate). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+201
to
+205
| this.interlaced = true; | ||
| Log.i(TAG, `Interlaced per codec metadata hint (${width}x${height})`); | ||
| this.onVerdict({ interlaced: true, algorithm: "bwdif", fieldOrder: this.fieldOrder }); | ||
| this.scheduleFieldOrderVote(); | ||
| } |
Comment on lines
+178
to
+181
| start(): void { | ||
| if (this.timer) return; | ||
| this.timer = window.setInterval(() => this.sample(), SAMPLE_INTERVAL_MS); | ||
| } |
Comment on lines
+1234
to
+1238
| "absolute inset-0 size-full min-h-0 min-w-0 object-fill", | ||
| visibleSlotId !== slotId && "invisible pointer-events-none", | ||
| // Deinterlaced output replaces the raw frames visually; keep the video | ||
| // composited (opacity, not visibility) so requestVideoFrameCallback keeps firing | ||
| visibleSlotId === slotId && deinterlaceActiveSlots[slotId] && "opacity-0", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds render-side automatic deinterlacing to the web player for interlaced IPTV streams (typically broadcast 1080i content). Deinterlacing runs as a WebGL2 overlay that intercepts each decoded frame via
requestVideoFrameCallback, draws the result to a per-slot canvas, and lets the<video>element continue driving playback and audio — the MSE pipeline is untouched.How it works
Detection — two paths:
frame_mbs_only_flag == 0in H.264 SPS orfield_seq_flag/interlaced_source_flagin H.265 VUI, it plumbs that hint from demuxer → remuxer → worker → playervideo-infoevent. The deinterlace pipeline activates immediately on the first frame without waiting for the heuristic warm-up.Field order (TFF/BFF): Consecutive frame pairs are compared under both TFF and BFF hypotheses by their temporal-midpoint interpolation error — the wrong assumption averages fields two field-times apart, so motion amplifies the error. A majority vote with abstention on still/ambiguous pairs drives the field-order decision; the renderer starts with TFF and re-emits if BFF wins.
Algorithm — bwdif (GLSL port of FFmpeg's
vf_bwdif): Per-pixel motion adaptation keeps both fields in still areas (full vertical resolution, no bob shimmer on static content like logos and subtitles) and reconstructs moving pixels with the edge-preserving spatio-temporal filter, clamped to the temporal neighbourhood. Runs one frame behind via a 3-frame texture ring at field rate for 50p motion output from 25i input. Chroma keeps a separate vertical low-pass to remove field-interleaved color combing from progressive 4:2:0 upsampling. The shader was reviewed againstlibavfilter/bwdifdsp.candvf_bwdif.cwith three gaps corrected (frame-boundary edge fallback, interpolation direction sign, spatial-check short-circuit on kept lines).Resolution gate: Active for all content up to 1080 (width ≤ 1920, height ≤ 1088), covering 480i, 576i, 720i, and 1080i. Disabled above 1080 regardless of content.
User control: A settings dropdown toggle (auto / off) is persisted to player storage.
Testing
devlab () gains three scan channels for exercising the detector:
Verified in Chrome against devlab: TFF channel tff=4/0, BFF channel bff=4/0, no false positive on 1080p, 2160p gated off.
Files changed
web-ui/src/mpegts/deinterlace/— new module: detector, renderer, bwdif algorithm, GL utilities, type registryweb-ui/src/mpegts/demux/— H.264/H.265 SPS parsing for interlace flagsweb-ui/src/mpegts/— plumbing through remuxer, worker messages, player coreweb-ui/src/components/player/— settings toggle, video-player hook integrationtools/devlab/— scan channels and README documentation🤖 Generated with Claude Code