The Free and Open Web Is Under Attack at the IETF (EFF)

An EFF Deeplinks post by tori-noble (June 2026) arguing that publishers and large technology firms are trying to bend IETF technical standards into a way to lock down automated access to publicly available web content — and that the same machinery aimed at AI crawlers would also choke off journalism, archiving, research, and accessibility tools.

The argument

EFF’s claim is that two IETF efforts, pitched as narrow technical fixes, carry a much broader gatekeeping payload:

• ai-preferences — a working group standardizing machine-readable “preference signals” (an evolution of robots.txt) that let a site declare its content off-limits for AI training or deployment. EFF’s worry is less the signal itself than what attaches to it: in some jurisdictions such a declaration could become legally binding, turning a voluntary convention into an enforceable access right.
• web-bot-auth — a working group on cryptographic bot identification. Sold as relief from aggressive bot traffic that strains servers, it also supplies the plumbing for selective, identity-keyed access — the technical precondition for licensing automated access and excluding whoever has not paid or been approved.

Put together, EFF argues, the two give site operators a standards-blessed mechanism to impose licensing requirements on automated access to public content — a gate where there is now an open door.

Why EFF says this matters

Crawling and scraping are not just AI’s appetite; they underpin comparison shopping, security vulnerability detection, search, web archiving, and accountability journalism. Restricting them through ostensibly neutral standards, the post argues, quietly prioritizes publisher and platform revenue over the civic uses of open information — and the people who lose access first are journalists, researchers, archivists, startups, and accessibility-tool developers, not the big AI labs who can strike licensing deals.

Reading / caveats

• Tier T2, advocacy POV. EFF is an expert digital-rights NGO reporting on live IETF work, but this is an advocacy piece stating one side of a contested standards debate. The counter-view — that sites have a legitimate interest in controlling crawling load and AI reuse of their content — is named only to be rebutted. Treat the mechanism description as reliable and the threat framing as one position.
• The IETF outputs described are in-progress working-group drafts, not ratified standards; their final shape (and any legal bindingness) is unsettled.

Connections

• venue: ietf — the standards body where this plays out
• standards under debate: ai-preferences, web-bot-auth
• publisher: electronic-frontier-foundation · author: tori-noble

Related

ai-governance · risk-based-regulation · china-ai-regulation · synthesis