Pay per crawl
Charging an AI crawler for each fetch. A site sets a price; a crawler that has not agreed to pay gets HTTP 402 Payment Required instead of the page. Cloudflare shipped it in July 2025 and made the surrounding tool, AI Crawl Control, generally available that August, which put a 402 response and a price field in front of every paid customer on the network.
How it is priced
Per Cloudflare's docs: the price is set per domain, the minimum is $0.001 USD per crawl, and the charge applies to each successful retrieval (an HTTP 200 for the content). Optional dynamic pricing allows different prices for different content, and the domain must be set "Visible" before the feature can be enabled. The original design was blunter still — one flat per-request price for the whole site. Re-read 2026-09-17: unchanged — per domain, $0.001 minimum, dynamic pricing optional, "Visible" prerequisite; the doc's own last-updated date is still 28 Jul 2026.
What it is worth
The arithmetic is the point. At the floor price a site earns $1 per thousand crawls. That is survivable for a publisher whose pages are re-fetched constantly and meaningless for a small site: the one-year report notes that over half of AI crawler traffic re-fetches unchanged pages, so the revenue tracks bot appetite rather than anything a site does. The model pays whoever is already being crawled hardest.
Two design limits follow:
- It prices the fetch, not the use. A page crawled once and then quoted in a million answers earns $0.001. This is why Cloudflare announced in July 2026 that it is moving toward pay per use — paying when content creates value — with Ceramic.ai and You.com as first partners, and why RSL separates pay-per-crawl from pay-per-inference and pay-per-training.
- It needs the crawler to cooperate. 402 is a refusal with a price attached; collection depends on the AI company choosing to pay rather than route around the block.
The leverage behind it
A toll only works on a resource someone can be kept out of, so the enforcement half matters more than the price half. From 2026-09-15 Cloudflare blocks mixed-use crawlers — bots that serve search, agents and training at once — from ad-supported pages by default, for new customers, new sites of existing customers, and all free-tier users. Search-only crawlers such as Google's are untouched, which preserves the traffic-sending crawl while making the training crawl negotiable.
Related
ai-content-monetization · x402 · crawl-to-refer-ratio · rsl-standard · cloudflare-agentic-internet-report · cloudflare-monetization-gateway · crawl-budget · llms-txt