iOS harness: stop tap_xy reporting phantom successes; add pixel sampling for colour - #80
Open
GenericJam wants to merge 4 commits into
Open
iOS harness: stop tap_xy reporting phantom successes; add pixel sampling for colour#80GenericJam wants to merge 4 commits into
GenericJam wants to merge 4 commits into
Conversation
…to view_tree tap_xy/2 returned ok whenever the platform input API accepted the event, never when the app reacted. On the simulator accessibilityActivate succeeds on a Box with on_tap while SwiftUI never runs the handler; on a physical device the injected IOHID touch is accepted for every coordinate and delivered for none. Both reported ok, so an agent driving a device could not tell a working tap from a no-op. Bump a process-wide counter from every send helper that routes a user-originated event into the BEAM, sample it around the injection, and poll for 300ms. ok now means the app demonstrably reacted; everything else is a typed error (no_view_at_point / no_element_at_point / no_effect). Hit-test up front on both paths, not just the device one. tap_xy moves to a dirty IO scheduler since it can now block. view_tree returned no colour at all, so a styling regression that dropped every Box background was invisible to the one introspection API meant to show what the device drew. Each node now carries bg_color and text_color read back off UIView/CALayer as 0xAARRGGBB ints — the repo's canonical colour representation (guides/theming.md). Documented the same keys on the Android bridge contract, which no shipped MobBridge.kt implements yet. Fixes a latent bug the new tests surfaced: :json.decode maps JSON null to :null, which normalize_view_tree passed straight through, so Android labels compared unequal to nil and a null children list would have crashed Enum.map. Docs state the real per-platform capability instead of the aspirational one, in Mob.Test, mob_nif.erl, CLAUDE.md and AGENTS.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds :mob_nif.ui_paint_debug/0, a census of which view/layer classes hold readable paint, and a :class field on every view_tree node so a caller can tell which renderer drew a node — and therefore why a colour is nil. Extends colour extraction to the layer tree (CALayer.backgroundColor, CAShapeLayer.fillColor, CATextLayer.foregroundColor). Measured on device that takes it from 2 of 443 nodes to 4 of 443: SwiftUI on iOS 26 paints through SDFLayer or rasterises into contents, so app content carries no readable paint property at all. See the decision record — colour verification needs screenshot sampling, not layer introspection.
view_tree colour is nil for virtually all SwiftUI content on iOS 26 (4 of 443 nodes when measured), so a styling regression like the glass theme that discarded every Box background was only findable by pixel-diffing screenshots by hand. Sampling pixels is the only reliable answer. New NIF sample_region/4 (iOS) returns raw RGBA for a cropped region, with the crop done inside the render so one element's pixels cross distribution instead of a framebuffer. It shares screenshot/3's capture path via the extracted mob_capture_window/mob_capture_image helpers. Mob.Test.sample_color/2 addresses a region by element :id (through element_frames) or an explicit rect, and reduce_rgba/3 is the pure reduction: average, dominant, dominant_share, distinct, pixels — because a card with text on it is not one flat colour and a bare mean of it is misleading either way. sample_region stays strictly #if !MOB_RELEASE rather than joining screenshot's MOB_ENABLE_SCREENSHOT opt-in: arbitrary-rect pixel reads reconstruct the screen region by region, which would silently grant the capability that opt-in exists to make deliberate. Pinned by a test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Formatting only. These lines came in with the view_tree colour work earlier on this branch (f1fb5e0, d04118c) and fail `xcrun clang-format --dry-run -Werror`, which the pre-commit checklist runs. Kept out of the sampling commit so that diff stays reviewable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The on-device test harness reported success for things that did not happen, and could not see colour at all. Both made agent-driven verification on iOS unreliable — an iOS renderer bug sat unverified in a downstream repo for weeks partly because of this.
1.
tap_xyreported:okwhen nothing happenedOn a physical iPhone it returned
:okfor every coordinate — including empty space — and nothing ever happened;mob_send_touch_phasereturnsYESfor API availability, never delivery. On the simulator, tapping a Box withon_tap:returned:okwhile the handler never ran, because SwiftUI gives a plain.onTapGestureno accessibility action.Now the return value reflects whether a handler actually ran: a process-wide atomic counter is bumped by every send helper that routes a user event into the BEAM (
mob_send_tap,mob_send_event,mob_send_change), sampled before injection and polled for 300 ms after.:ok{:error, :no_view_at_point}{:error, :no_effect}{:error, :no_element_at_point}@docnow states the real per-platform capability, including that coordinate tapping does not work on a physical device and that the counter only sees Mob's own handlers (sidecar caveat).tap_xyalso moved toERL_NIF_DIRTY_JOB_IO_BOUND— it was already sleeping on a normal scheduler.Device-verified: a theme chip (Box +
on_tap) now returns{:error, :no_effect}where it previously returned a false:ok.2.
view_treecolour: measured, then abandoned as designedReading
UIView.backgroundColor/UILabel.textColorgave colour for 2 of 443 nodes. Also walkingCALayer/CAShapeLayer/CATextLayerreached 4 of 443. The newui_paint_debug/0census says why:SwiftUI on iOS 26 paints via
SDFLayeror rasterises intocontents, exposing no readable paint property. Layer introspection is a dead end for SwiftUI content — recorded in an ADR rather than iterated on further. The colour fields stay (correct for UIKit chrome, root background, future Android) alongside a new:classfield so a caller can tell which renderer drew a node and therefore why a colour isnil.3.
sample_color/2— colour verification that workssample_region/4returns raw RGBA for a natively cropped region (small payload over distribution), reduced by a pure, unit-tested Elixir function toaverage,dominant,dominant_shareanddistinct— because a card with text on it is not one flat colour, and the share/distinct fields tell you whether to trustdominant.Gated strictly
#if !MOB_RELEASE, deliberately not the|| defined(MOB_ENABLE_SCREENSHOT)its neighbour uses: arbitrary-rect pixel reads let a caller reconstruct the screen region by region, which would hand release builds the capability that opt-in exists to gate. Pinned by a test on the guard string.Acceptance test, on device: two chips,
background: :primaryvs:surface_raised, under a glass theme.0xFF20202E0xFF20202E0xFF4000AF0xFF101023The regression that previously required hand pixel-diffing is now
assert a.dominant != b.dominant.Also reported, not fixed
scroll_info,scroll_toandelement_framesare behind#if !MOB_RELEASEon iOS while Android registers them unconditionally, so an iOS release build raisesnif_error(not_loaded)and kills the calling screen process. None of the three touches private API — thescreenshotopt-in pattern would fit.Verification
1011 tests passing,
format/credo --strict/erlfmt/clang-formatclean; ObjC compiled and run on an iPhone 17 simulator; non-vacuity of new tests proven by sabotage-and-restore.🤖 Generated with Claude Code