Content Independence Day, one year on (Cloudflare)
Cloudflare's July 2026 report on the year since it began blocking AI crawlers by default. It is the network operator's own measurement of who crawls the web and what comes back, and it is the closest thing this spoke holds to a supply-side count of the traffic collapse the demand-side sources (pew-ai-overviews-clicks, great-content-no-longer-works) argue about.
What it measures
- AI training crawling reached 52% of all crawler requests by June 2026, up from 22% in spring 2025. Pure search crawling — the kind that can return a click — is "a small and declining share."
- Mixed-use crawlers are over 36% of crawler activity: one bot fetching for search, agents and training at once, so a site cannot allow the crawl that sends traffic without also allowing the crawl that does not. Cloudflare says it intends to drive mixed use to zero within a year.
- Non-human traffic crossed 50% of all internet traffic for the first time.
- Human traffic to the most-crawled content categories fell by up to 40% in under a year. Of every hour spent searching, Cloudflare puts 15 minutes on the open web; the rest stays in the assistant.
- More than 50 publisher–AI licensing agreements have been signed since 2023 — the whole market, and it is a list of large publishers.
The sentence the report turns on: "Content is still being crawled, indexed, and used — but increasingly without corresponding traffic being returned to the source."
The argument
Cloudflare frames a four-step path from measurement to money — transparency (site owners see which bots take what), scarcity (default blocking makes access refusable), leverage (a refusable resource can be priced), then a licensing economy. pay-per-crawl and the Monetization Gateway are the third and fourth steps shipped as products.
The step that does the work is scarcity, and it is not a market outcome — it is a default setting on one CDN. That is also the caveat: the report is published by the company selling the tollbooth, using data only it can see. The measurements are first-party and unaudited, and the conclusion they support is that publishers should buy Cloudflare's answer.
Why it matters here
The spoke has documented the click collapse from the marketer's side for a year — ai-search-shift, seo-no-longer-drives-growth, the halved click-through in pew-ai-overviews-clicks. This source supplies the other half of the same event: the crawl continues, at growing volume, and the return path is gone. That gap is what ai-content-monetization is about, and crawl-to-refer-ratio is how it gets counted.
Related
ai-content-monetization · pay-per-crawl · crawl-to-refer-ratio · cloudflare-monetization-gateway · cloudflare · crawl-budget · rsl-standard