Mobile device agent

An LLM agent whose tool surface is a phone: it reads the device's UI state, then taps, swipes, types and launches apps to carry out an instruction given in words. The mobile case of GUI automation — the agent operates the same interface a person would, because on a phone there usually is no API to call instead.

Entered this spoke on 2026-08-03 with mobilerun; the corner is one source deep.

The defining choice: tree or pixels

Two ways to give the model the screen.

• Accessibility tree — the OS already maintains a structured description of what is on screen for screen readers: element roles, labels, bounds. Read that and the model gets semantics for a few hundred tokens.
• Screenshot — send the pixels and let vision do the work. Universal, and the only option when the surface is canvas-drawn, a game, or an app that never bothered with accessibility labels.

mobilerun builds on the tree and keeps vision as a fallback, claiming a 500× smaller payload and a ~7-point AndroidWorld lead over screenshot-first rivals. Generalized: an agent does better against a surface that describes itself than against one it has to look at, so the first question for any GUI agent is what structured description the platform already exposes. Accessibility APIs exist because of disabled users; agents inherited the affordance.

Shape

The deployed shape splits device and brain: a device-side helper that exposes control (Portal in mobilerun) and an operator-side runtime holding the loop, model and tools. The phone runs no model, which is what makes these frameworks model-agnostic — the same substrate-independence harnesses show on the coding side.

That was one source's shape, and 2026-08-30 supplied the counterexample. seekerclaw runs the runtime on the phone — a Node.js process inside an Android app, a foreground service with a heartbeat and a cron timer — and still calls a remote model. So the split above is really two splits and only one of them is doing the work: the model is remote; the harness need not be. Move the loop onto the device and the agent stops being a session someone starts from a laptop and becomes resident, which is what makes chat channels (Telegram, Discord) its interface rather than a CLI.

The two also point the phone in opposite directions. For mobilerun the device is the target — the thing being tapped. For seekerclaw it is the host, and device control is one tool module of thirteen beside memory, skills, MCP, web fetch and scheduling. Worth keeping both under this page rather than splitting it: the corner is defined by the phone being in the loop at all, and the interesting variable is which end of the loop it sits at.

Longer tasks pull in agent-orchestration: a planner decomposes the goal, an executor performs the taps. Same manager-executor topology, a different end effector.

Why it sits here

The agentic-tooling thesis is that the value moved to what wraps the model. Mobile is that argument at its starkest — the model contributes nothing device-specific, and everything that makes the agent work (the accessibility bridge, the tree encoding, the gesture tools, the planner) is harness. It is also the spoke's first surface that is neither a repo, a browser nor a chat channel.

Adjacent but distinct: an agentic browser's "act on my behalf" mode (../web-browsers-wiki) is the same problem inside one application; ../osint-wiki holds the profile-building agents that would be an obvious consumer of phone control. Neither owns the device layer.

Open

• Verification. A tap is not a diff — there is no git diff to review before the agent sends the message or spends the money. The loop-engineering feedback signal and reversibility bound are unanswered here. Partly answered 2026-08-30 by seekerclaw, and the answer is not verification. Facing the hardest version of this — an irreversible Solana transaction — it does not try to make the action reviewable. It bounds the loss: per-transaction and daily caps in USDC and SOL, preflighted before any send, with the agent signing silently under cap and falling back to a wallet popup over it. Reads are bounded by money and writes by consent (x402 GET settles silently, POST always asks). That answers "spends the money" and leaves "sends the message" exactly where it was — a cap is denominated in currency and a sent message has no denomination.
• Corroboration. Two sources now, and still nobody outside either project has measured anything: mobilerun's AndroidWorld lead is DroidRun's own run, and seekerclaw publishes no benchmark at all. See agent-evals.

Related

mobilerun · droidrun · text-first-agent-design · agentic-tooling · agentic-coding-harness · agent-orchestration · context-rot · agent-guardrails · agent-evals · seekerclaw