llms.txt
llms.txt is a proposed standard — a markdown file at /llms.txt that gives LLMs/AI tools a
“curated overview” of a site’s key content, “to provide LLM-friendly content” at inference time. It’s
a concrete, mechanism-level answer to part of the spoke’s GEO
question (how do you actually make a site usable by AI?). Source: llmstxt.org.
The problem it addresses
“Context windows are too small to handle most websites in their entirety,” and “converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is both difficult and imprecise.” llms.txt offers a hand-curated, link-and-summary markdown map instead.
The actual format
The spec (llmstxt-spec) is tighter than “a markdown file” suggests. It was proposed by Jeremy
Howard (fast.ai) on September 3, 2024, and prescribes a fixed order: an H1 with the site/project
name (the only required section), a blockquote summary, optional free-form markdown detail, then
H2-delimited sections holding link lists. Each list entry is “a required markdown hyperlink
[name](url), then optionally a : and notes about the file.” A reserved Optional H2 marks links
that “can be skipped if a shorter context is needed” — a built-in priority lever for tight context
windows. Notably the spec does not define an llms-full.txt; that’s a community convention, not part
of the proposal.
vs. robots.txt / sitemaps
A useful contrast for the spoke: robots.txt controls crawler access; sitemaps list all
pages; llms.txt is a curated, inference-time overview sized to fit an LLM context. It’s the GEO
counterpart to those older files — though, like everything in generative-engine-optimization, its
real payoff is unmeasured (adoption by AI vendors is voluntary and uneven) — keep it in the
“plausible mechanism, no neutral benchmark yet” bucket the synthesis uses. The strongest signal yet is
negative and first-party: Google’s own AI-optimization guidance states “Google Search itself doesn’t use
them” — creating an llms.txt won’t help or harm Search visibility (google-ai-optimization-guide).
So for the largest engine it’s confirmed inert; its value, if any, rests on the other AI vendors.
Related
llmstxt-spec · google-ai-optimization-guide · generative-engine-optimization · answer-engine-optimization · ai-search-shift · brand-depth-ai-recommendations · customer-success-ai-readable-proof