A peer-reviewed study tested the top 10,000 sites. Three quarters of the blocks that stopped a headless browser fell to correcting HTTP headers — no browser modifications required.
Most deployed bot detection is header inspection. Researchers at the University of Bamberg — best paper at ARES 2026 — tested the Tranco top 10,000 sites with 40,000 visits and found headless Chromium blocked on 15.2% of sites versus 6.8–7.2% for headed configurations. Then the finding that matters: about 75% of the blocks that applied to headless only were bypassed by correcting HTTP headers alone — mainly the sec-ch-ua client hints that announce HeadlessChrome.
That is not an argument that bot detection is fake. It is an argument that a lot of what is sold as detection is a string comparison wearing a machine-learning label.
| Measure | Result | Why it matters |
|---|---|---|
| Sites blocking headless Chromium | 15.2% | Roughly one in seven of the largest sites blocks default automation. |
| Sites blocking headed browsers | 6.8–7.2% | The gap between these two numbers is the headless signal. |
| Sites blocking headless only | 772 | Against just 14 that blocked only headed — detection is overwhelmingly aimed at headless. |
| Headless-only blocks defeated by header fixes | ~75% | Fix the client hints and most of those sites let you through. |
Sites probing navigator.webdriver |
34% | The single most common check, and trivially spoofable in most automation stacks. |
| Blocks attributable to four commercial vendors | 82% | What "blocked" means on the web is largely defined by a handful of products. |
Tested against the Tranco top 10,000. Full methodology and the authors' signal taxonomy are in the paper, linked below.
There are two genuinely different products being sold under the same words, and buyers rarely get told which one they are looking at.
| Header inspection | Client detection | |
|---|---|---|
| Reads | What the request claims | What the client is |
| Defeated by | Editing headers — minutes of work | Replicating browser internals convincingly, at scale |
| Cost to operate | Nearly nothing | Real per-request work |
| Catches | Default automation, which is most of it | Default automation plus anything that bothered to look |
| Honest description | Filtering | Detection |
Header inspection is not worthless — it stops the overwhelming majority of automated traffic, which never bothers to disguise itself. The problem is only that it is sold as detection, and that a buyer protecting something valuable has no way to tell the difference from the outside.
Separate 2026 work on detecting AI agents approached the problem from the opposite direction: rather than enumerating browser properties, it asked which observable features actually separate human, bot and agent sessions. Working from 6,025 sessions of behavioural event streams, the strongest discriminators were mouse event rate and the standard deviation of click duration. The timing features everyone expected to matter — the pause-think-act pattern of an LLM-driven agent — did not survive feature selection.
That is the more durable direction, and there is a reason for it. Properties are static facts a client reports about itself, and every one of them can be patched before the page loads. Behaviour is an accumulation of what the client did over time, and faking it convincingly across thousands of sessions is a genuinely different order of problem from setting a flag.
It is also why single-signal detection fails in both directions. No one property separates real from automated users — which is the same reason the honest answer to "will this block a real customer?" is a rate, not a yes.
None of these require a lab, and a vendor who cannot answer them has not measured the thing they are selling.
1. What proportion of your detection is derivable from HTTP headers alone? This is the Bamberg question. If the answer is "most of it", you are buying a filter.
2. What is your measured false-positive rate on real traffic? Not "we don't block real users" — a number. A false positive on a residential connection is the cost that never appears on a pricing page, and it is the one most likely to hurt you.
3. Can you show me the signals behind a specific verdict? If you cannot see why a request was blocked, you cannot tell a real customer from a false positive when they complain.
4. What happens to my site when your service is unreachable? Fail-open is the right default for most sites and a liability for a few — but only one of those is a considered answer, and the other is a discovery you make during an outage.
Where ZeroBot stands, briefly. Detection runs as an HTTP-level API call that returns the verdict and the reasons behind it, so question 3 is answered by design rather than by policy. Fail-open is the default, in both the hosted API and the self-hosted script. Browser fingerprinting is available as a separate, optional layer for sites that need it. We publish no false-positive rate yet, because we have not run that measurement to a standard we would defend — which is itself an answer to question 2, and a more honest one than a number we invented.
Testing the Tranco top 10,000 sites with 40,000 visits, the researchers found headless Chromium blocked on 15.2% of sites while headed configurations were blocked on only 6.8-7.2%. Critically, about 75% of the blocks that applied to headless only were bypassed simply by correcting HTTP headers — the client hints and User-Agent string that gave away "HeadlessChrome".
No, and that is an overread the paper does not make. It means a large share of deployed detection is header inspection, which is cheap and effective against default automation but is not the same thing as detecting automation. The remaining 25% used deeper methods that a header change did not defeat.
Of the top 10,000, about 34% probed navigator.webdriver and 46% probed at least one automation signal. 82% of the blocks observed came from four commercial bot-detection vendors, which means the behaviour of a handful of products largely defines what "blocked" means on the web.
Peer-reviewed work on behavioural detection found the strongest discriminators were mouse event rate and the standard deviation of click duration — behavioural measures — rather than the LLM timing patterns people expected to matter. Behaviour is expensive to fake convincingly at scale, which is why it holds up better than static properties.
Ask four things: what proportion of your detection is derivable from HTTP headers alone; what is your measured false-positive rate on real traffic; can you show me the signals behind a specific verdict; and what happens to my site when your service is unreachable. A vendor who cannot answer the first and second has not measured them.
University of Bamberg — Detecting Bot Detection (ARES 2026, best paper) · arXiv — What does it take to detect an AI agent? Minimal feature sets · arXiv — Web price extraction, state of the art · cside — detecting AI agents and stealth browsers. The stealth-tooling trend figures are as reported by the sources above; treat vendor-published detection rates as marketing until independently reproduced.
Per-request scoring, the signals returned to you, fail-open by default. $50/mo flat.
Get started free →