Skip to content

iOS harness: stop tap_xy reporting phantom successes; add pixel sampling for colour - #80

Open
GenericJam wants to merge 4 commits into
masterfrom
fix/ios-test-harness-honesty
Open

iOS harness: stop tap_xy reporting phantom successes; add pixel sampling for colour#80
GenericJam wants to merge 4 commits into
masterfrom
fix/ios-test-harness-honesty

Conversation

@GenericJam

Copy link
Copy Markdown
Owner

The on-device test harness reported success for things that did not happen, and could not see colour at all. Both made agent-driven verification on iOS unreliable — an iOS renderer bug sat unverified in a downstream repo for weeks partly because of this.

1. tap_xy reported :ok when nothing happened

On a physical iPhone it returned :ok for every coordinate — including empty space — and nothing ever happened; mob_send_touch_phase returns YES for API availability, never delivery. On the simulator, tapping a Box with on_tap: returned :ok while the handler never ran, because SwiftUI gives a plain .onTapGesture no accessibility action.

Now the return value reflects whether a handler actually ran: a process-wide atomic counter is bumped by every send helper that routes a user event into the BEAM (mob_send_tap, mob_send_event, mob_send_change), sampled before injection and polled for 300 ms after.

Value When
:ok an event reached the BEAM within 300 ms
{:error, :no_view_at_point} hit-test found nothing (now on both platforms)
{:error, :no_effect} input accepted, no handler ran
{:error, :no_element_at_point} simulator: view present, no AX element

@doc now states the real per-platform capability, including that coordinate tapping does not work on a physical device and that the counter only sees Mob's own handlers (sidecar caveat). tap_xy also moved to ERL_NIF_DIRTY_JOB_IO_BOUND — it was already sleeping on a normal scheduler.

Device-verified: a theme chip (Box + on_tap) now returns {:error, :no_effect} where it previously returned a false :ok.

2. view_tree colour: measured, then abandoned as designed

Reading UIView.backgroundColor/UILabel.textColor gave colour for 2 of 443 nodes. Also walking CALayer/CAShapeLayer/CATextLayer reached 4 of 443. The new ui_paint_debug/0 census says why:

204x SwiftUI._UIInheritedView / CALayer            all paint props 0
 64x SwiftUI._UIInheritedView / SwiftUI.SDFLayer   all paint props 0
 14x _UIInheritedView / CGDrawingLayer             all 0, contents=14

SwiftUI on iOS 26 paints via SDFLayer or rasterises into contents, exposing no readable paint property. Layer introspection is a dead end for SwiftUI content — recorded in an ADR rather than iterated on further. The colour fields stay (correct for UIKit chrome, root background, future Android) alongside a new :class field so a caller can tell which renderer drew a node and therefore why a colour is nil.

3. sample_color/2 — colour verification that works

sample_region/4 returns raw RGBA for a natively cropped region (small payload over distribution), reduced by a pure, unit-tested Elixir function to average, dominant, dominant_share and distinct — because a card with text on it is not one flat colour, and the share/distinct fields tell you whether to trust dominant.

Gated strictly #if !MOB_RELEASE, deliberately not the || defined(MOB_ENABLE_SCREENSHOT) its neighbour uses: arbitrary-rect pixel reads let a caller reconstruct the screen region by region, which would hand release builds the capability that opt-in exists to gate. Pinned by a test on the guard string.

Acceptance test, on device: two chips, background: :primary vs :surface_raised, under a glass theme.

Build active inactive
without the glass fix 0xFF20202E 0xFF20202E bug reproduced
with #78 0xFF4000AF 0xFF101023 fix confirmed

The regression that previously required hand pixel-diffing is now assert a.dominant != b.dominant.

Also reported, not fixed

scroll_info, scroll_to and element_frames are behind #if !MOB_RELEASE on iOS while Android registers them unconditionally, so an iOS release build raises nif_error(not_loaded) and kills the calling screen process. None of the three touches private API — the screenshot opt-in pattern would fit.

Verification

1011 tests passing, format/credo --strict/erlfmt/clang-format clean; ObjC compiled and run on an iPhone 17 simulator; non-vacuity of new tests proven by sabotage-and-restore.

🤖 Generated with Claude Code

GenericJam and others added 4 commits August 9, 2026 20:14
…to view_tree

tap_xy/2 returned ok whenever the platform input API accepted the event,
never when the app reacted. On the simulator accessibilityActivate
succeeds on a Box with on_tap while SwiftUI never runs the handler; on a
physical device the injected IOHID touch is accepted for every
coordinate and delivered for none. Both reported ok, so an agent driving
a device could not tell a working tap from a no-op.

Bump a process-wide counter from every send helper that routes a
user-originated event into the BEAM, sample it around the injection, and
poll for 300ms. ok now means the app demonstrably reacted; everything
else is a typed error (no_view_at_point / no_element_at_point /
no_effect). Hit-test up front on both paths, not just the device one.
tap_xy moves to a dirty IO scheduler since it can now block.

view_tree returned no colour at all, so a styling regression that
dropped every Box background was invisible to the one introspection API
meant to show what the device drew. Each node now carries bg_color and
text_color read back off UIView/CALayer as 0xAARRGGBB ints — the repo's
canonical colour representation (guides/theming.md). Documented the same
keys on the Android bridge contract, which no shipped MobBridge.kt
implements yet.

Fixes a latent bug the new tests surfaced: :json.decode maps JSON null to
:null, which normalize_view_tree passed straight through, so Android
labels compared unequal to nil and a null children list would have
crashed Enum.map.

Docs state the real per-platform capability instead of the aspirational
one, in Mob.Test, mob_nif.erl, CLAUDE.md and AGENTS.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds :mob_nif.ui_paint_debug/0, a census of which view/layer classes hold
readable paint, and a :class field on every view_tree node so a caller can
tell which renderer drew a node — and therefore why a colour is nil.

Extends colour extraction to the layer tree (CALayer.backgroundColor,
CAShapeLayer.fillColor, CATextLayer.foregroundColor). Measured on device
that takes it from 2 of 443 nodes to 4 of 443: SwiftUI on iOS 26 paints
through SDFLayer or rasterises into contents, so app content carries no
readable paint property at all. See the decision record — colour
verification needs screenshot sampling, not layer introspection.
view_tree colour is nil for virtually all SwiftUI content on iOS 26 (4 of
443 nodes when measured), so a styling regression like the glass theme that
discarded every Box background was only findable by pixel-diffing screenshots
by hand. Sampling pixels is the only reliable answer.

New NIF sample_region/4 (iOS) returns raw RGBA for a cropped region, with the
crop done inside the render so one element's pixels cross distribution instead
of a framebuffer. It shares screenshot/3's capture path via the extracted
mob_capture_window/mob_capture_image helpers.

Mob.Test.sample_color/2 addresses a region by element :id (through
element_frames) or an explicit rect, and reduce_rgba/3 is the pure reduction:
average, dominant, dominant_share, distinct, pixels — because a card with text
on it is not one flat colour and a bare mean of it is misleading either way.

sample_region stays strictly #if !MOB_RELEASE rather than joining screenshot's
MOB_ENABLE_SCREENSHOT opt-in: arbitrary-rect pixel reads reconstruct the screen
region by region, which would silently grant the capability that opt-in exists
to make deliberate. Pinned by a test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Formatting only. These lines came in with the view_tree colour work earlier on
this branch (f1fb5e0, d04118c) and fail `xcrun clang-format --dry-run -Werror`,
which the pre-commit checklist runs. Kept out of the sampling commit so that
diff stays reviewable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant