Inblog
Start your company blog
GEO

Pay Per Crawl

Pay Per Crawl is Cloudflare's billing model that lets websites charge an AI crawler per request for access to their content. It is the first large-scale use of the long-dormant HTTP status code 402 (Payment Required), launched in private beta on July 1, 2025.

Pay Per Crawl is Cloudflare's billing model that lets websites charge an AI crawler per request for access to their content. It is the first large-scale use of the long-dormant HTTP status code 402 (Payment Required), launched in private beta on July 1, 2025.

Why It Matters

Search engine crawling has always rested on an implicit exchange: content is taken, traffic is sent back. AI crawling broke that exchange. According to analysis of Cloudflare Radar data for Q1 2026, Anthropic's crawler fetched roughly 24,000 pages for every single visitor it referred back, and OpenAI roughly 1,300—against Googlebot's ratio of about 5:1. For publishers left uncompensated by AI referral traffic, the only lever used to be blocking. Pay Per Crawl adds a new one: block, or monetize.

How It Works

  1. When a crawler requests a protected page, the server responds with a 402 and a header stating the per-request price.
  2. If the crawler retries with a payment-intent header, it receives the content with a 200 OK. Crawlers can also declare a maximum price upfront and get content immediately when the cost fits.
  3. Publishers set one flat per-request price across the domain and choose one of three options per crawler: Allow (free), Charge, or Block.

Cloudflare acts as Merchant of Record for settlement, and a companion standard called Web Bot Auth verifies crawler identity with cryptographic signatures so bots cannot spoof a friendlier identity. If robots.txt is a polite request with no enforcement, Pay Per Crawl is a tollbooth enforced at the network level.

Where Things Stand

On the same day it announced Pay Per Crawl, Cloudflare began blocking AI crawlers by default for newly onboarded domains—establishing a "blocked by default, paid by choice" posture. The capability has since been folded into the AI Crawl Control product, and custom 402 responses are now available to every paid Cloudflare customer. In February 2026, Stack Overflow publicly adopted Pay Per Crawl with Cloudflare, reporting that many unauthorized crawlers simply stopped trying once they hit 402 responses. Whether per-crawl payments become a meaningful revenue line is still being tested, but the model is widely seen as the technical infrastructure for content-compensation negotiations between AI companies and publishers.

Sources:

How inblog Helps

Whether blocking, charging, or allowing is the right call depends on what your content is for. If your blog exists to win brand exposure inside AI answers, staying crawlable and earning citations usually beats locking crawlers out. inblog's built-in analytics show referral traffic from ChatGPT and other AI channels, so you can see whether AI crawling is actually paying you back in visits—and set your access policy based on data rather than guesswork.