Roughly 40% of top sites now try to block AI crawlers in robots.txt. Around 30% of AI scrapes ignore the file entirely. It is worth being clear about which of those two numbers your content strategy actually depends on.
robots.txt does not block anything. It is a voluntary convention — a machine-readable statement of what you would prefer crawlers to do. Compliant crawlers honour it because they have decided to. A Tollbit analysis found roughly 30% of AI scrapes ignore it. If your content strategy depends on that file being obeyed, it depends on the goodwill of the party you are trying to stop.
The 2026 measurement studies are remarkably consistent, and they tell a more interesting story than "blocking works" or "blocking fails".
| Finding | Figure | What it means |
|---|---|---|
| Top sites naming an AI crawler | 12.7% | Explicitly blocking by name is still a minority behaviour. |
| Effectively blocking AI crawlers | ~40% | Most of that comes from blanket wildcard rules rather than AI-specific decisions. |
| Explicitly allowing AI crawlers | 1.8% | Almost nobody opts in on purpose. |
| AI scrapes that ignore robots.txt | ~30% | The enforcement gap, and the reason this debate keeps returning. |
| News sites blocking Google-Extended that still appear in AI citations | 92.3% | Blocking the crawler is not the same as being absent from the answer. |
| News portals blocking GPTBot (DACH) | 40.4% | Against 2.27% in the long tail — an 18× gap between who can afford to care and who cannot. |
Sources: Zyte's State of Web Access 2026 (11,100 landing pages), Tollbit, BuzzStream, and an Ahrefs study of 24,550 DACH domains. Figures linked at the foot of this page.
Most of the confusion in this debate comes from treating a policy and a control as the same thing, when they operate in completely different places.
| A policy (robots.txt) | A control (what runs in front) | |
|---|---|---|
| Enforced by | The crawler | You |
| Applies to | Crawlers that agree | Every request, agreed or not |
| Fails | Open — a new crawler is allowed by default | Closed — a new crawler is blocked until you say otherwise |
| Tells you anything | No — there is no log of who read it | Yes — you see what arrived and what you did |
| Right for | Signalling intent to cooperative parties | Enforcing a decision |
The practical consequence: if you have written a robots.txt rule and stopped there, you have published a preference. That is a legitimate thing to do — but it is not a control, and nobody should be measuring its effectiveness as though it were one.
The most significant publisher shift of 2026 was the move from blocklist to allowlist. Reuters and Time both switched to blocking AI crawlers by default and permitting only approved ones, following People Inc. and The Atlantic. One publisher reportedly went from blocking around 2,100 user agents to over 30,000 by inverting the model.
The reason to prefer it is not that it catches more — it is the direction of the default. With a blocklist, every crawler nobody has heard of is permitted until someone notices. With an allowlist, it is blocked until you make a decision. For content being harvested by infrastructure that changes monthly, failing closed is the only version that stays accurate without maintenance.
The honest caveat is that allowlists fail closed on your side too. A crawler you meant to permit and forgot is blocked, and you will hear about it from a partner rather than a log.
The single most important technique in this area is also the least used: do not trust the User-Agent. Anyone can send Googlebot as a string. Almost nobody can control the reverse DNS of the address they are connecting from.
So the check is two-step, and both steps matter:
The same principle applies to the newer generation of agents. A request that self-identifies as an AI assistant's fetcher is making a claim, not presenting a credential. Without a signature you can verify, it is a User-Agent with better branding.
Keep the robots.txt. It costs nothing, cooperative crawlers respect it, and it is the correct way to signal intent to the parties willing to listen.
Stop treating it as the enforcement layer. If a decision matters — licensing, paywalling, protecting a catalogue — enforce it where the request arrives, not in a file the requester chooses to read.
Verify by reverse DNS, not by claim. Allowlist what you have verified. Everything else is a client that has not proven anything.
Be honest with yourself about the cost. Blocking has a price in legitimate traffic and in citation visibility, and it is not always effective. Blocking the crawler and being absent from the answer are different outcomes, and only one of them is under your control.
Where ZeroBot fits: the same API check described on the stop scrapers page runs before your content is served, so it applies whether or not the client agreed to read your robots.txt. It scores the client — including whether the claimed identity matches the one that is actually connecting.
No. It is a voluntary convention with no enforcement mechanism. Compliant crawlers honour it because they choose to; a Tollbit analysis found roughly 30% of AI scrapes ignore it entirely. It is a statement of intent, not an access control.
Something in front of your content that does not ask permission: verified-crawler allowlisting at the edge, a WAF rule, a rate limit, or an application-level check that scores the request before it reaches your content. The distinction is that these run regardless of what the client claims about itself.
Reverse DNS, not the User-Agent string. A genuine Googlebot resolves back to a Google-owned hostname, and you confirm that hostname resolves forward to the same IP. Anyone can send a User-Agent that says Googlebot; almost nobody can control the reverse DNS of their address.
For publishers, increasingly yes, and that is the direction the industry moved in 2026 — Reuters and Time both switched to blocking AI crawlers by default and allowing only approved ones. An allowlist fails closed: a new crawler you have never heard of is blocked until you decide otherwise, rather than permitted until you notice it.
Some legitimate traffic, and some search visibility. One paper associates aggressive blocking with roughly 7% weekly traffic loss, and blocking is not always effective anyway — analysis of top news sites blocking Google-Extended found over 92% of them still appearing in AI citations.
Zyte — State of Web Access 2026: AI agents · Zyte — four in ten sites block AI bots · Search Engine Journal — news sites default to blocking · Digiday — Reuters and Time adopt allowlists · WebProNews — publishers and Congress · Ahrefs — DACH AI bot block rates. The Tollbit figure and the citation analysis are as reported in the coverage above.
One API call, before your content is served. $50/mo flat, free tier.
Get started free →