Here is the bit the by-hand crowd will appreciate: the rate limit on the allowed classes is keyed per vendor, not per client address. This sounds like a detail until you have watched a single assistant prompt fan out across dozens of source addresses, each of them politely under any per-address cap you could write, while the aggregate quietly saturates your PHP workers. One shared key per vendor caps that vendor’s total rate across every address and every site on the box, and gives each vendor a bucket of its own, so no single company can starve the others’ allowance either. A throttled request, by the way, gets the same silent closed connection as a hard block, so in the logs the two are told apart by behaviour and not by status: a hard guard refuses every request of its class, a rate limit lets requests through at the allowed pace and refuses only the excess of a burst. Worth knowing before you conclude from three lines of log that a vendor has been “banned”.
Training and bulk-collection crawlers, the ones which sweep whole sites to feed a model and send no visitor back: blocked by default, with a per-site opt-in. AI search and citation indexers, the ones which make your site turn up in an AI answer: allowed by default and rate-limited, and so are the honest assistant fetchers, the ones that arrive because an actual person asked an assistant to read your page. An evasive fetcher, nominally user-triggered, which ignores no-crawl rules and, when blocked, drops its name and rotates addresses (the docs name one well-known assistant here, and so did the company which de-listed it as a verified bot): blocked by default, per-site opt-in. And nobody has ever complained about the read-aloud, notebook and ad-preview tools, so that small utility class stays open too, rate-limited like the rest.
All of this happens at your server’s own front door, in the web server itself, before a single line of Drupal or Backdrop runs, before any PHP worker is bothered and long before the database hears a word about it; a crawler refused there costs you roughly a string comparison and a closed connection, while one that gets deep enough to render pages has already burned your fuel just to be told no, and when the flood is the point, that difference is the whole economy of the defence. The door is yours, too: there is no third-party bot-defence service anywhere in this chain and no rules living in a console you cannot read, the maps and the limits ship with the platform itself, run on the same box that serves your sites, and are tuned as a part of its ordinary updates by the people who watch the vendors’ crawler documentation and the logs of a whole fleet of such boxes, so when the crawlers change their manners, and they change them constantly, we keep changing the rosters with them. Underneath this layer sit two more that do not go by declared names alone, judging behaviour and address instead, and behind them the system firewall holds the bans they produce; and a site parked behind Cloudflare is not left out of any of it, because the server recovers the real visitor address from the edge and keeps judging the visitor rather than the middleman.
Back to the blog »
There is one more thing you can do, and it is the opposite of blocking: you can tell the AI tools what your policy is. The emerging convention is a plain-text file served at /llms.txt, and every site here already has the route wired; the file is served straight from your site’s own files directory, so publishing a policy is uploading a text file over SFTP, no module and no configuration, with a per-domain variant for multi-domain sites, and if you would rather manage it from inside Drupal, the contrib module for it works over the same route, whichever file exists gets served.
We got a fair number of surprised messages since the policy landed, mostly of the “why is GPTBot getting a closed door on my site, I never asked for that” kind, which is perfectly fair, because who reads release notes, and the answer deserves more than a link. So here is what the machine already does for every Drupal, Backdrop or Pressflow site you host on it, why the defaults sit where they sit, and then how to move any of them for one site, without asking anyone.
There is nothing clever about changing the policy for one site. In your own account there is a small text file, one line per site, the site name followed by the flags you want: allow training for this one, block search for that one, allow the evasive fetcher on the research site, block the assistant fetchers on the members-only shop, in any combination and in any order, with comments allowed so the file can explain itself in six months. The machine picks the change up on its own within about two minutes, no reload to schedule, no ticket to raise, and deleting a site’s line puts it back on the defaults. The whole thing is your decision, in a file you own, on a server which keeps no opinion about your content that you cannot overrule.
Protection has a second edge, and the by-hand crowd knows this one painfully well: a rule that lets a scraper through costs you nothing you will ever notice, while a rule that catches a legitimate crawler costs you something you will be noticing for weeks, because to a busy rule the fetcher that collects the little icon shown beside your site in the search results looks exactly like the thin, odd traffic the rule was written to stop. So the same machinery that blocks also keeps the whitelists: addresses of the legitimate crawlers are refreshed from the vendors’ own published lists, Google’s special-case crawlers included, precisely so our own anti-abuse rules don’t eat your favicon out from the search results; a false positive that breaks legitimate traffic is the one bug we treat as worse than the abuse it was meant to stop.
Three things stay on regardless, and there is no line you can add to change them, because they belong to hygiene rather than to policy: the outright scrapers and download tools, the mass-harvesting bad bots; requests which arrive wearing an opt-out token as their identity, and this one is a small joy, because those tokens are robots.txt directives, a real crawler never sends them as a live user-agent, so anything claiming to be one has just confessed to a forgery, and is refused with zero false positives (the one place that phrase is honest!); and the credential probes, the requests fishing for your .env and .git and their cousins. If one of those ever catches a genuine request of yours, that is a conversation with your host, the same as any other server-level control.
Do you know what your platform decided, on your behalf, about whether an AI company may keep a copy of everything you have ever written, and would anybody have told you either way? Or did you once open a night’s access log of your Drupal site, find that one crawler had eaten more bandwidth than every human visitor put together, and then spend the evening arguing with a robots.txt which nothing on the other end had the slightest intention of reading?
So your content is yours again, and it cost one line in one file you own, or no line at all, if the defaults suit you, which for most sites they do. There is a quieter question sitting behind all of this: when the visitor turned away is not a crawler at all but an actual person on the wrong side of a very confident shield, who lets them back in, and how long does it take? The policy, the flags and the file live in the AI crawlers guide, with the edge-side view next door for the curious, which explains the maps in enough detail that you may end up writing your own, and that is roughly how we got here in the first place.
First, the sorting. AI traffic is not one thing, and treating it as one thing was the mistake of the earlier single switch which blocked every AI user-agent it knew about, search fetchers included, which is a bit like fencing off the shop because one visitor kept photographing the shelves. The current policy sorts the visitors by what they came for, keyed on a distinctive token in the user-agent rather than a vendor name, so that a training crawler and the assistant fetcher of the same company land in different classes with opposite defaults (a bare vendor name, on its own, matches nothing at all, on purpose).
Then the honest edges, which on this subject matter more than the features. A user-agent is a string, and any client can claim to be an allowed assistant to land in an allowed class; the robust hardening, gating each allowed class on the vendor’s published signed address ranges, is deliberately not in this cut, and even if it ever arrives it could not make the evasive fetcher blockable by address, since that one rotates outside its own published range. The class maps are a roster, current as of the cut which shipped them, and a fetcher whose token is not in the roster yet is not blocked by this layer at all; it falls through to ordinary handling, because the maps act only on agents they recognise, fail-open by design rather than fail-closed. And the evasive block is best-effort by definition: once an agent abandons its user-agent it looks like a browser, and a browser is not this policy’s problem. The backstop for the rotating, undeclared, genuinely abusive traffic is the intrusion-detection layer under this one, which reads behaviour rather than declarations and takes no request’s word for what it is, so the two layers accomodate each other rather well: the maps sort the honest visitors, the detector deals with the rest. None of this is a caveat we would rather you did not notice; it is the reason the policy can be trusted, since a shield which claims to be absolute is the one you should worry about.
The balance in that list is the whole point, and it answers the fear behind most of the surprised messages: blocking the training class does not make a site disappear from AI tools. Indexing stays open, the person-triggered fetch stays open, so you keep the visibility and skip the bulk harvesting, which is roughly the deal you already have with the search engines, only spelled out.
