Sunday, 20 July 2025
28.4 C
Singapore
29.8 C
Thailand
17.1 C
Indonesia
28.4 C
Philippines

Cloudflare blocks AI crawlers by default to protect online content

Cloudflare blocks AI crawlers by default to protect online content, giving creators more control and a new "Pay Per Crawl" option.

You’re now better protected online, as Cloudflare moves to block AI bots from scraping websites without permission. The internet infrastructure company announced on July 1 that AI web crawlers will be blocked by default across its platform. This change is part of a broader effort to help content creators control how their work is used, especially by artificial intelligence companies that may access and use online content for training without approval.

AI access will now require your permission

If you host your website through Cloudflare, you’ll automatically block known AI bots unless you choose otherwise. When you set up a new domain, you’ll be asked whether you want to allow these crawlers access to your site. This means you’re no longer at risk of having your content scraped by AI models without awareness.

Cloudflare is testing a new “Pay Per Crawl” programme with selected publishers and creators to give you more control. You can set a fee for AI companies that want to access your content through this. The companies can then decide whether to pay the fee or look elsewhere. While it’s not yet open to all users, Cloudflare says this trial ensures that content is used “correctly—with permission and compensation.”

More tools to protect your website

You may recall that Cloudflare began letting users block AI crawlers in 2023. At the time, it only affected bots that respected your website’s robots.txt file — an instruction manual for crawlers that isn’t legally enforceable. But not all bots play fair.

So, later that year, Cloudflare introduced a way to block all AI bots — even those that ignore your robots.txt settings. That setting is now turned on by default for new Cloudflare customers. If you’ve been concerned about AI scraping, this gives you extra safety from the start.

Cloudflare uses its own list of known AI crawlers to spot which bots to block. On top of that, they launched a quirky new feature in March: an “AI Labyrinth.” When an unauthorised AI crawler tries to access a site, it gets lost in a maze of redirects and dead ends, keeping it away from your content.

Major publishers support Cloudflare’s efforts

Several well-known companies, including The Associated Press, Fortune, The Atlantic, Stack Overflow, and Quora, have already signed on to Cloudflare’s anti-crawler initiative. These websites, like yours, are concerned about the shift in how people find information online. Users are increasingly turning to AI chatbots instead of traditional search engines.

“People trust the AI more over the last six months, which means they’re not reading original content,” Cloudflare CEO Matthew Prince said at a recent Axios Live event. That shift could hurt publishers, especially if their content is used without credit or payment.

Cloudflare is working with AI companies to identify and verify their crawlers to address this. The goal is to ensure these bots clearly explain their purpose, whether for training models, answering questions, or powering search tools. Once you know what a crawler wants your content for, you can decide whether to allow it.

Prince said, “Original content is what makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.”

Hot this week

Honor opens largest store in Singapore with hands-on access to Magic V5 foldable

Honor opens its biggest store in Singapore at Plaza Singapura, featuring the Magic V5 and exclusive launch-day promotions.

WeTransfer clears up AI concerns after backlash over new terms

WeTransfer clarifies that it won’t use your files to train AI, following backlash over its updated terms of service that raised user concerns.

Juniper Networks: Southeast Asia accelerates shift towards AI-native self-driving networks

AI-native networking is helping Southeast Asian enterprises move from reactive IT to proactive innovation, improving scale, efficiency, and service.

YouTube Premium Lite launches in Singapore with ad-free access for S$7.98 a month

YouTube Premium Lite launches in Singapore, offering most ad-free videos for S$7.98 monthly with a phased rollout in the coming weeks.

Confluent Cloud now listed under new AI category on AWS Marketplace

Confluent Cloud is now available in AWS Marketplace's AI Agents and Tools category, supporting real-time data access for AI applications.

Xi Jinping raises concerns over China’s overinvestment in the EV and AI sectors

Xi Jinping warns against China's overinvestment in EVs and AI, urging officials to prioritise sustainability over short-term growth.

Meta refuses to sign the EU’s AI code of practice, citing legal concerns

Meta declines to sign the EU's AI Code of Practice, citing legal concerns and excessive requirements that exceed the scope of the AI Act.

Microsoft closes Movies & TV store on Xbox and Windows without warning

Microsoft has shut down its Movies & TV store on Xbox and Windows, ending new sales but maintaining access to previously purchased content.

Honor opens largest store in Singapore with hands-on access to Magic V5 foldable

Honor opens its biggest store in Singapore at Plaza Singapura, featuring the Magic V5 and exclusive launch-day promotions.

Related Articles

Popular Categories