Crawl-to-refer ratio
How many pages a platform takes for every visitor it sends back. Cloudflare Radar
computes it as HTML requests from a platform's crawler user-agents divided by HTML requests whose
Referer header points at that platform, normalised to one referral. It is the closest thing the
open web has to a price of being crawled, and the reason the monetization question in
ai-content-monetization exists at all.
The numbers
From Cloudflare Radar, first week of August 2025 (whole network, unfiltered): Anthropic ≈50,000:1, OpenAI ≈887:1, Perplexity ≈118:1. A week in June 2025 put Anthropic at 70,900:1 and Mistral at 0.1:1 — a spread of six orders of magnitude between platforms, and a wide swing week to week for the same one.
The ratio tightens by industry. For News & Publications: Anthropic 2,500:1, OpenAI 152:1, Perplexity 32.7:1. For Computer & Electronics: Anthropic 8,800:1 down to Perplexity 88:1. So the sites whose content is most valuable to a model are also the ones getting the least bad exchange rate, which is faint comfort at 2,500 fetches per visit.
Two structural facts sit under the ratios: training was about 80% of AI bot crawling in July 2025, with user actions and undeclared purposes under 5% combined, and by June 2026 AI training crawling was 52% of all crawler traffic (cloudflare-agentic-internet-report). Traffic-returning crawl is the minority case and shrinking.
Reading it honestly
Cloudflare's own caveat: native app traffic carries no Referer header, so referrals from an
assistant's mobile app are invisible and the ratios overstate the gap by an unknown amount. The figures
also come from one network, computed by the company selling the response to them, and they move
enough week to week that any single number is a snapshot rather than a rate.
What survives the caveats is the ordering, which is stable across every cut: search-purpose crawlers send traffic, training crawlers do not, and the gap between them is measured in thousands rather than percentages.
Related
ai-content-monetization · pay-per-crawl · cloudflare-agentic-internet-report · crawl-budget · pew-ai-overviews-clicks · ai-search-shift · cloudflare