Cloudflare Unveils ‘Pay per Crawl’ Marketplace to Let Websites Charge AI Bots for Content Access

Cloudflare, the internet infrastructure giant that powers around 20% of global web traffic, has launched a bold new experiment — a marketplace allowing website owners to charge AI companies for crawling their content.

Dubbed “Pay per Crawl,” the platform is now in private beta and aims to reshape the way AI companies access and use online content. The move comes amid growing concerns from publishers about unauthorized scraping by AI bots, which has become increasingly common with the rise of large language models and AI search tools.

How Pay per Crawl Works

Participating website owners can decide whether to:

• Charge AI bots a micropayment each time they crawl their content,
• Allow free access,
• Or block them entirely.

The platform offers detailed analytics so publishers can see which bots are accessing their content and for what purpose — whether it’s for AI training, search responses, or other use cases.

To join, both publishers and AI companies must be on Cloudflare. Each side can then set their own crawl rates, while Cloudflare acts as the intermediary, facilitating payments and handling compliance. Notably, no cryptocurrency or stablecoins are involved at this stage, despite industry speculation.

A Shift Toward Content Control

This launch reflects Cloudflare’s push for a "permission-based approach" to web crawling. Starting July 1, all new websites onboarded to Cloudflare will automatically block AI crawlers by default, unless the site owner opts to allow access.

Major publishers like TIME, The Associated Press, The Atlantic, ADWEEK, Fortune, and Condé Nast have already joined Cloudflare’s initiative to block AI bots unless authorized.

Historically, websites allowed search engines like Google to crawl them in exchange for traffic referrals. But with the rise of AI chatbots and agents delivering direct answers to users — often without links — publishers are seeing diminishing returns on that bargain.

Cloudflare’s internal data underscores the problem:

• Google’s crawler scrapes 14 times for every referral it sends.
• OpenAI’s crawler scrapes 17,000 times per referral.
• Anthropic scrapes a staggering 73,000 times for each referral.

The Bigger Vision: Programmatic Access in an “Agentic” Future

Cloudflare sees “Pay per Crawl” as a stepping stone to a future powered by AI agents. These intelligent tools would automatically query the web on behalf of users — for example, to synthesize research or generate custom recommendations — and pay site owners to access high-quality content.

Imagine assigning a budget to an AI assistant to gather medical studies or restaurant reviews — with those funds programmatically distributed to content owners through Cloudflare’s network.

Will It Work?

The concept is promising but faces significant hurdles. Convincing AI companies — which currently scrape for free — to start paying is a tall order. At the same time, smaller publishers may struggle to get enough visibility in the marketplace to earn meaningful revenue.

Still, Cloudflare, with its deep ties to both tech companies and media organizations, may be uniquely positioned to broker this new relationship between content creators and AI firms.

Whether “Pay per Crawl” becomes the future of content licensing or just a stepping stone remains to be seen — but the battle over AI access to the internet’s data is clearly just getting started.