Skip to content

Fix describe remote web content discovery and add regression tests#901

Open
lapfelix wants to merge 1 commit intofacebook:mainfrom
lapfelix:codex/webview-describe-remote-content-tests
Open

Fix describe remote web content discovery and add regression tests#901
lapfelix wants to merge 1 commit intofacebook:mainfrom
lapfelix:codex/webview-describe-remote-content-tests

Conversation

@lapfelix
Copy link
Copy Markdown

Motivation

idb ui describe was not reliably returning accessibility elements inside embedded web content (WebView), even when those elements were present and VoiceOver-accessible.

Root causes:

  • The describe path did not enable remote-content discovery options.
  • Remote discovery could be starved when native traversal reported near-total frame coverage (common with full-screen proxy/container elements), because covered points were skipped and same/frontmost PID hits were filtered out.
  • Coverage accounting included AXWeb* container roles, which could overstate “already covered” area and suppress remote probing.

Implemented changes:

  • CompanionLib/FBIDBCommandExecutor.m: enabled remote discovery for describe calls by setting collectFrameCoverage and default remoteContentOptions.
  • FBSimulatorControl/Commands/FBSimulatorAccessibilityCommands.m: excluded web container roles (AXWeb* / Web*) from base frame coverage; added near-full-coverage fallback probing (coverage >= 0.98); allowed seen/frontmost/unknown PID hits in that fallback path to recover hidden content; skipped near-fullscreen proxy hits to avoid duplicate wrappers; kept explicit unsigned index conversions needed for strict warning-as-error builds.
  • FBSimulatorControlTests/Utilities/AccessibilityDoubles.h and FBSimulatorControlTests/Utilities/AccessibilityDoubles.m: added translation-to-element mapping support in the translator double so tests can model frontmost-tree vs hit-tested remote elements.
  • FBSimulatorControlTests/Tests/Unit/FBSimulatorAccessibilityCommandsTests.m: added targeted regression tests:
    • testCoverageCalculationSkipsWebContentRoleElements
    • testRemoteDiscoveryFindsHiddenContentWhenCoverageIsNearFull
    • testRemoteDiscoverySkipsFullscreenProxyInNearFullCoverageMode
  • FBSimulatorControlTests/Tests/Unit/FBSimulatorAccessibilityCommandsTests.m: retained test-only int64_t casts needed to satisfy strict sign-compare checks in this toolchain.

Compatibility:

  • No gRPC interface changes.
  • No CLI schema/flag changes.
  • Behavior changes are scoped to accessibility describe discovery and supporting tests.

Test Plan

  1. Build tests target:

    xcodebuild -project FBSimulatorControl.xcodeproj -target FBSimulatorControlTests -configuration Debug -sdk macosx build
  2. Run accessibility unit suite:

    export DYLD_FRAMEWORK_PATH=$PWD/build/Debug
    xcrun xctest -XCTest FBSimulatorControlTestCase,FBSimulatorAccessibilityCommandsTests $PWD/build/Debug/FBSimulatorControlTests.xctest

    Result: 21 tests, 0 failures (includes all new regression tests).

  3. Build companion libraries:

    xcodebuild -project FBSimulatorControl.xcodeproj -target CompanionLib -configuration Debug -sdk macosx build
  4. Manual runtime verification on booted simulator:

    build/Build/Products/Debug/idb_companion --udid 78F1F8EA-CDE6-45A6-BB20-C5D40241D8E6 --grpc-port 11601 --log-level debug
    idb --companion 127.0.0.1:11601 ui describe-all --json

    Verified output includes embedded webview labels/element.

Related PRs

None.

I accepted the CLA

@meta-cla meta-cla bot added the CLA Signed label Feb 12, 2026
@kzlar
Copy link
Copy Markdown

kzlar commented Feb 20, 2026

@lapfelix This is exactly what I've been looking for! Appreciate the work on this, it works well for me. Really hope it could be merged into the main branch.

Have you looked into interacting with the "Sign in with Apple" modals? idb seems pretty unreliable in describing them and I'm wondering if it could be improved in a similar way?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants