Developers block entire countries to stop AI crawlers. There's a better move.

By early 2025, AI crawler traffic had gotten bad enough that developers stopped fine-tuning robots.txt and started blocking entire countries outright. That's not a hypothetical, that's an actual engineering decision real teams made under real production pressure. Ars Technica reported it in March 2025, and the discussion racked up 360 points and 275 comments on Hacker News, the kind of engagement that only happens when a pain point is widely shared, not niche.

The pain that started this

The complaint behind that Ars Technica story was simple: AI crawlers were hitting sites hard enough, and indiscriminately enough, that some teams found it easier to block traffic by geography than to fight crawler-by-crawler. A year later, the traffic has only grown. Cloudflare Radar data shows overall AI and search-crawler traffic up 18% from May 2024 to May 2025, and AI bot traffic on the user-driven axis, the requests triggered by someone actually asking an AI a question, up 15-fold over the same window.

ChatGPT-User requests specifically surged 2,825% in that period and reached 1.3% of all crawler share by mid-2025. None of that is training-crawler noise. It's real users triggering real fetches, which is exactly the traffic a blanket block also throws away with everything else.

The AI crawlers ratios that made people snap

That bars chart is scaled to OpenAI's peak, 3,700 pages crawled for every one referral sent back, because the raw numbers don't fit on a normal 0-100 scale and that's the point. PerplexityBot peaked lower, above 700 to 1 in the same window, still a rate no engineer would accept from a paying customer, let alone a bot that pays nothing.

Compare that to Googlebot, which as of January 2026 still covers 1.7 to 167 times more unique URLs than any individual AI crawler, GPTBot, ClaudeBot, or PerplexityBot, depending on which one you compare it to. Developers were being crawled hard by bots that referred almost nothing back, while getting none of the coverage benefit Google's own crawler provides. You'd block that too.

Why blocking AI crawlers backfires

Here's the part that gets lost in the understandable anger: the crawler that trains a model and the crawler that fetches a page to answer a live question are often different bots from the same company. Block them together, by country or by user agent, and you don't just stop training scrapes. You stop ChatGPT, Claude, or Perplexity from fetching your page at the moment a user's question could have cited it, the same live-fetch failure mode we cover in AI agents can't find your price. A country block doesn't know the difference. It just sees a user agent string and says no.

So the frustration is real, and the instinct behind it is understandable. I don't think any of those developers were wrong to be angry, or wrong to reach for the bluntest tool available to them at the time. The tool just wasn't precise enough for the actual problem they were trying to solve. A country block is a sledgehammer, built for a job that needed a scalpel. What most sites actually needed, and mostly didn't have until recently, was a meter.

There's a second-order cost to the country-block approach that rarely makes it into the initial decision. Blocking by geography doesn't just stop bots, it stops human visitors in those regions who happen to share an IP range or network path with flagged crawler traffic, an increasingly common side effect as more AI infrastructure gets hosted on the same cloud provider ranges real users connect through. A rule built to stop crawler load can end up quietly rejecting legitimate customers in the same block, and most teams don't find out until support tickets start piling up from a region that suddenly can't reach the site.

The 2026 alternative: charge instead of block

Cloudflare's Pay Per Crawl, generally available since August 2025 and made free across every plan tier during Agents Week in April 2026, lets a site charge AI crawlers per fetch instead of blocking them outright. A crawler that won't pay gets a 402 Payment Required instead of your content. Cloudflare says customers on the feature are now sending over a billion of those 402 responses a day across the network.

Condé Nast, Time, the Associated Press, BuzzFeed, Reddit, and Pinterest have all adopted AI Crawl Control. Stack Overflow published a joint case study with Cloudflare on its use of the feature in February 2026. That's not a fringe list. It's a signal that the choice is no longer binary between open access and a blanket block.

The underlying shift explains why metering beats blocking right now. OpenAI's search indexer coverage jumped from 4.7% of sites in April 2025 to over 55% by mid-2025, while its training-crawler coverage actually dropped from 84% to 12% over the same window. The bots doing the live-answer fetching and the bots doing the model training are diverging fast, which means a single blanket rule aimed at 'AI bots' is targeting a category that's splitting in two right in front of you. Meter the one, and you keep your shot at the citation from the other.

This is not a niche mistake. Our own benchmark work across 60 enterprise sites, published in the enterprise GEO readiness framework, found 71% of enterprise sites block at least one major AI crawler without realizing it, usually as collateral damage from a security rule, a bot-management default, or exactly the kind of blanket geography block this piece opened with. The country-blocking instinct doesn't just show up in developer forums. It's already sitting, unnoticed, in the majority of enterprise stacks we've audited.

Pricing matters less than the mechanism here. Whatever a site charges per crawl is almost beside the point for most publishers, since the real value isn't the per-fetch revenue, it's the leverage. A crawler that has to pay, even a token amount, has an incentive to fetch efficiently instead of hammering a site indiscriminately, which is the actual behavior developers were reacting to in the first place. The billion-a-day 402 figure isn't a billion dollars changing hands. It's a billion moments where a crawler chose not to pay rather than fetch content it didn't have a clear reason to want, which is precisely the filtering a blanket block was trying, clumsily, to achieve.

What to actually do this week

The bot-taxonomy point is worth dwelling on, because it's the actual technical fix underneath all of this. Cloudflare's own crawler categories split search-and-retrieval bots, OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User, from foundation-model training bots, GPTBot, ClaudeBot, CCBot, Google-Extended. A rule written against 'AI crawlers' as one undifferentiated category can't distinguish between them. A rule written against the specific user-agent strings in each category can allow, meter, or block each independently, which is the granularity a blanket geography block never had in the first place.

None of this is free of tradeoffs, and it's worth saying plainly: metering isn't a guaranteed win for every site. A small publisher with limited engineering resources may reasonably decide that the complexity of bot-by-bot rules isn't worth it yet, and a simple, coarser block is the pragmatic choice until traffic justifies the setup. The point isn't that everyone needs Pay Per Crawl by Friday. It's that a country block used to be the only option, and now it's the worst one available to anyone with the resources to do better.

Pull your own server logs and separate training crawlers from live-fetch crawlers before you touch robots.txt again, the exact first step in the technical audits we run for clients. Then decide, bot by bot, whether you want to meter it, allow it, or block it, instead of reaching for a country-level rule because it's the only lever that felt available. If you're already running GEO work to earn citations, a blanket block undoes it before it starts, and it's worth checking your current robots.txt against the same audit we walk through in llms.txt, explained, since the two files should agree with each other, not fight.

Revisit the decision quarterly, not once. Crawler behavior is moving fast enough, 15x growth in a year on the user-driven axis alone, that a bot-management policy set today can be stale within two quarters. Treat it the same way you'd treat any other infrastructure config that sits in front of real revenue: owned, monitored, and reviewed on a calendar, not set once during a security sprint and forgotten.

See where you are cited today

A free snapshot audit of your rankings and AI citations before we ever talk.