Text-first agent design

Don’t hand the agent the surface. Hand it a derived representation chosen for meaning per token, and go look at the raw thing only where the representation runs out.

The pattern shows up whenever an agent meets a medium that is expensive to serialize — a phone screen, an hour of footage, a repo too big for the window — and it is consistently the design decision that separates the fast, accurate tool from the slow, hallucinating one.

Four instances, three orgs, one day

mobilerun claims 500× smaller payloads and a ~7-point AndroidWorld lead from reading the accessibility tree instead of screenshots. video-use compresses an entire shoot into a markdown file and never sweeps the timeline. watch-skill spends its frame budget on distinct content. None of them is compressing information for its own sake; each found a representation the platform or the task already implies, and stopped rendering pixels the model would only have to decode back into meaning.

Note what the two video tools do differently: watch-skill keeps a real frame budget because it answers questions about what a video shows; video-use cuts on speech boundaries, so what a video says is nearly the whole signal. The representation follows the task, not the medium.

The neighbouring move: the representation the model knows

open-edit reads a per-word transcript like video-use does, then diverges at the output end: the agent authors caption designs in HTML and CSS, which a binary renders without a browser. The input side is this pattern; the authoring side is a different question with the same shape — not “what description does the platform maintain” but “what notation is the model already fluent in.” paperclip makes the same bet on the other side of the hub, replacing per-database APIs with a virtual file system because file operations are what models handle well.

Both moves pick a representation for the model’s sake rather than the medium’s, and the failure modes differ: a derived input representation can silently omit something, while a familiar authoring notation mostly fails visibly, in the render. Nobody has measured either.

Why it works

A screenshot makes the model re-derive semantics that already existed upstream — a labelled button was a labelled button before it was rendered. Text that came from the platform’s own model of the surface arrives pre-interpreted, cheaper, and without recognition error. Accessibility APIs exist because of disabled users; agents inherited the affordance.

The economics are the context-rot argument: what enters the window determines output quality, so the win is in curation, not in a bigger window. It also composes with the spoke’s other token levers as a distinct move — compress (headroom), remove (safe-prompt-pruning), re-encode (this), price-arbitrage (pxpipe).

The inverse case, which sharpens rather than breaks it

pxpipe runs the other way: it renders text into PNGs to exploit image-token pricing on a coding harness. That looks contradictory and isn’t. A coding harness has no upstream structured description to fall back on — text is already the most meaningful form available, so the only lever left is what the tokens cost. Mobile and video both ship a structured description for free.

So the rule isn’t “prefer text.” It’s ask what description the platform already maintains before deciding what to feed the model — and if the answer is nothing, you’re down to pricing tricks.

It doesn’t need an agent

md-this-page is the pattern with the machinery removed: a browser extension where Mozilla’s Readability heuristic drops the ads and chrome, Turndown emits Markdown, and a person pastes it into a chat. No proxy, no MCP, no model deciding what to cut. That it works at all is the useful part — much of what bloats a context window is structural boilerplate that a 2010-era Reader Mode heuristic can strip by rule, which is the same bet safe-prompt-pruning makes with deterministic passes. The expensive tools are automating a judgment that is often not a judgment.

Open

• Nobody has measured where the fallback threshold sits. The three agent-driven tools keep a raw-medium escape hatch and none publishes how often it fires or what it costs when the representation silently omits something. The failure mode is quiet: a button with no accessibility label, an edit decision that hinged on a facial expression.
• Every number here is self-reported. See agent-evals.

Related

mobilerun · video-use · open-edit · watch-skill · md-this-page · pxpipe · context-rot · headroom · safe-prompt-pruning · mobile-device-agent · paperclip · agentic-tooling