A playbook for deciding who gets your content: what to log, how to verify a claim, where policy belongs, and why a self-declared name is not an identity.
Enforce at the edge, verify rather than trust, and scope by path. Those three rules cover most of what goes wrong. Everything below is the detail: what to record so you can answer questions later, how to turn a claimed identity into a verified one, and where in your stack the decision should actually be made.
The reasoning behind the first rule is on robots.txt is a request, not a control.
Log the full User-Agent string, the path, the source address, and the result of any verification you performed. Aggregating on the claimed name alone produces a report of what clients say about themselves, which is not the same as what arrived. Keep the raw value even when it is empty — an absent header is a finding, and you cannot tell a stripped header from an omitted one unless you kept the evidence.
A self-declared crawler name is worth nothing on its own. Verification is two lookups, and both have to pass:
A single site-wide policy forces a bad compromise: strict enough for checkout means hostile to browsing, and lenient enough for browsing means weak where it counts. Split it. Static assets pass untouched. Public content follows your crawler policy. Search, login, registration and checkout get the tightest scoring you can justify — those are the endpoints where automation has something to gain.
The cheapest request to reject is the one you never rendered. Putting the decision in your templating or controller layer means you have already spent the database query, the session write and the render. A Worker, a reverse-proxy rule or the first lines of your application all cost a fraction of that.
Every rule you add is a hypothesis about who is real. The only thing that confirms or refutes it is reading the blocked list. Put a recurring appointment on it, because a false positive is invisible from the inside: the visitor who could not check out simply did not become a customer, and nothing in your metrics tells you they were there.
Where ZeroBot fits. Step 4 is the API call: one HTTP request that returns a verdict before your application renders, so nothing expensive happens for traffic you are going to refuse. Step 1 is the response — the score plus the individual signals behind it, so you keep the evidence on your side rather than only in a vendor dashboard.
Trusting the name. The most common configuration error in this area is
allowlisting Googlebot as a string. It takes seconds to test and grants access to
anyone who typed it.
Blocking where your customers are. Residential and mobile address space is where real people are. A blocklist that catches an attacker by catching a neighbourhood is an outage with extra steps — see why IP reputation stopped working.
Never reading the blocked list. Without review you cannot tell a working rule from a quietly expensive one, and the cost of the second never appears in a report.
At the point the request arrives, before your application renders content — a Worker, a reverse proxy, or the first lines of your application. Enforcing inside your content pipeline means you have already paid the cost of generating the response you are about to withhold.
By verified identity. A name is a string the client chose; verification is a fact you established. Practically that means reverse DNS for named search crawlers and, where a vendor publishes one, their official IP range list — refreshed, because these change.
You should. Treating a blog and a checkout identically is how sites end up either too strict for browsing or too loose for the endpoints that matter. Crawl policy for content, tighter scoring for search, login, registration and checkout, and let static assets through entirely.
Log the full User-Agent, the verified identity where you can establish it, the path, and the address. Aggregating by claimed identity alone tells you what clients say about themselves; aggregating by verified identity tells you what actually arrived.
They are the reason enforcement has to exist. robots.txt, self-declared names and unverified claims are all optional for the client. Rate limits, verified allowlisting and per-request scoring apply whether or not the client cooperates.
One API call, the signals returned to you, fail-open by default. $50/mo flat.
Get started free →