Mobile device agent

An LLM agent whose tool surface is a phone: it reads the device’s UI state, then taps, swipes, types and launches apps to carry out an instruction given in words. The mobile case of GUI automation — the agent operates the same interface a person would, because on a phone there usually is no API to call instead.

Entered this spoke on 2026-08-03 with mobilerun; the corner is one source deep.

The defining choice: tree or pixels

Two ways to give the model the screen.

• Accessibility tree — the OS already maintains a structured description of what is on screen for screen readers: element roles, labels, bounds. Read that and the model gets semantics for a few hundred tokens.
• Screenshot — send the pixels and let vision do the work. Universal, and the only option when the surface is canvas-drawn, a game, or an app that never bothered with accessibility labels.

mobilerun builds on the tree and keeps vision as a fallback, claiming a 500× smaller payload and a ~7-point AndroidWorld lead over screenshot-first rivals. Generalized: an agent does better against a surface that describes itself than against one it has to look at, so the first question for any GUI agent is what structured description the platform already exposes. Accessibility APIs exist because of disabled users; agents inherited the affordance.

Shape

The deployed shape splits device and brain: a device-side helper that exposes control (Portal in mobilerun) and an operator-side runtime holding the loop, model and tools. The phone runs no model, which is what makes these frameworks model-agnostic — the same substrate-independence harnesses show on the coding side.

Longer tasks pull in agent-orchestration: a planner decomposes the goal, an executor performs the taps. Same manager-executor topology, a different end effector.

Why it sits here

The agentic-tooling thesis is that the value moved to what wraps the model. Mobile is that argument at its starkest — the model contributes nothing device-specific, and everything that makes the agent work (the accessibility bridge, the tree encoding, the gesture tools, the planner) is harness. It is also the spoke’s first surface that is neither a repo, a browser nor a chat channel.

Adjacent but distinct: an agentic browser’s “act on my behalf” mode (../web-browsers-wiki) is the same problem inside one application; ../osint-wiki holds the profile-building agents that would be an obvious consumer of phone control. Neither owns the device layer.

Open

• Verification. A tap is not a diff — there is no git diff to review before the agent sends the message or spends the money. The loop-engineering feedback signal and reversibility bound are unanswered here.
• Corroboration. One source, one vendor’s benchmark. See agent-evals.

Related

mobilerun · droidrun · text-first-agent-design · agentic-tooling · agentic-coding-harness · agent-orchestration · context-rot · agent-guardrails · agent-evals