Cloudflare blocks AI crawlers by default to protect online content

You’re now better protected online, as Cloudflare moves to block AI bots from scraping websites without permission. The internet infrastructure company announced on July 1 that AI web crawlers will be blocked by default across its platform. This change is part of a broader effort to help content creators control how their work is used, especially by artificial intelligence companies that may access and use online content for training without approval.

AI access will now require your permission

If you host your website through Cloudflare, you’ll automatically block known AI bots unless you choose otherwise. When you set up a new domain, you’ll be asked whether you want to allow these crawlers access to your site. This means you’re no longer at risk of having your content scraped by AI models without awareness.

Cloudflare is testing a new “Pay Per Crawl” programme with selected publishers and creators to give you more control. You can set a fee for AI companies that want to access your content through this. The companies can then decide whether to pay the fee or look elsewhere. While it’s not yet open to all users, Cloudflare says this trial ensures that content is used “correctly—with permission and compensation.”

More tools to protect your website

You may recall that Cloudflare began letting users block AI crawlers in 2023. At the time, it only affected bots that respected your website’s robots.txt file — an instruction manual for crawlers that isn’t legally enforceable. But not all bots play fair.

So, later that year, Cloudflare introduced a way to block all AI bots — even those that ignore your robots.txt settings. That setting is now turned on by default for new Cloudflare customers. If you’ve been concerned about AI scraping, this gives you extra safety from the start.

Cloudflare uses its own list of known AI crawlers to spot which bots to block. On top of that, they launched a quirky new feature in March: an “AI Labyrinth.” When an unauthorised AI crawler tries to access a site, it gets lost in a maze of redirects and dead ends, keeping it away from your content.

Major publishers support Cloudflare’s efforts

Several well-known companies, including The Associated Press, Fortune, The Atlantic, Stack Overflow, and Quora, have already signed on to Cloudflare’s anti-crawler initiative. These websites, like yours, are concerned about the shift in how people find information online. Users are increasingly turning to AI chatbots instead of traditional search engines.

“People trust the AI more over the last six months, which means they’re not reading original content,” Cloudflare CEO Matthew Prince said at a recent Axios Live event. That shift could hurt publishers, especially if their content is used without credit or payment.

Cloudflare is working with AI companies to identify and verify their crawlers to address this. The goal is to ensure these bots clearly explain their purpose, whether for training models, answering questions, or powering search tools. Once you know what a crawler wants your content for, you can decide whether to allow it.

Prince said, “Original content is what makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.”