Agent-browser equivalent for desktop and mobile apps

On Android the hard constraint is input injection, not observation. You can see the screen a few ways - Accessibility Service (view tree, no pixels needed), MediaProjection (pixels, but the user must consent to each capture session), or ADB/UIAutomator (dev setup). To actually tap and swipe inside another app you need one of: AccessibilityService.dispatchGesture(), ADB `input tap`, or root. And Play policy only allows the Accessibility API for accessibility purposes, which is why most cross-app automation ships as a dev tool over ADB rather than as a store app.

The practical shapes: - ADB-over-WiFi + UIAutomator: what Appium and Maestro build on; reliable, but needs pairing and a dev-ish setup. - Accessibility service: what Tasker-style automation uses; no PC needed, but Play-policy sensitive and OEM ROMs kill these services aggressively. - Vision loop (screenshot -> model -> coordinates -> gesture): most general, works on games and canvas UIs, but slow and still needs the same input path.

iOS is similar but has no MediaProjection equivalent for third-party apps - cross-app control is XCUITest/WebDriverAgent over a dev connection, which is why mobile agents mostly start on Android.

Curious what agent-device uses underneath; the input side is where most of these projects hit the wall.

The accessibility-tree approach makes a lot of sense for keeping token usage down.

How do you handle apps where the accessibility tree is incomplete though? Custom controls/canvas-heavy UIs can expose very little semantic information. Do you fall back to vision in those cases, and if so how do you keep the recorded flow deterministic enough to replay?