A new licensing framework has been introduced to help web publishers set clear rules on how artificial intelligence (AI) companies can use their content. On 11 September, major online platforms, including Reddit, Yahoo, Medium, Quora, and People Inc., announced support for Really Simple Licensing (RSL). This open content licensing system allows site owners to outline how AI developers should pay for access to their data.
The RSL Standard builds on the long-standing robots.txt protocol, which allows publishers to tell web crawlers which parts of their site they can and cannot access. Unlike the older system, however, RSL enables publishers to add specific licensing and royalty conditions. These can be placed directly into a robots.txt file or embedded within books, videos, and training datasets that companies may wish to monetise.
The initiative is being led by the RSL Collective, a newly formed rights organisation. It is headed by Eckart Walther, co-creator of the Really Simple Syndication (RSS) standard and former CardSpring chief executive, and Doug Leeds, former head of IAC Publishing and Ask.com. “The goal is to create a new, scalable business model for the web,” Walther told in an interview “RSL takes some of those early RSS ideas and creates a new layer for the entire internet where licensing rights and compensation rights are defined.”
Multiple payment models for AI use
The RSL Standard supports a variety of licensing structures, from free access to paid models. Website owners can choose to charge AI companies a subscription or apply a pay-per-crawl fee, requiring payment each time a bot visits their site. Another option is a pay-per-inference model, where publishers are compensated whenever an AI system draws on their content to generate a response. Crawlers operating for other reasons, such as archiving or inclusion in search engines, can continue to function without change.
Some media outlets, including Vox Media, News Corp, and The New York Times, have already signed individual licensing agreements with companies like OpenAI and Amazon. However, the RSL Collective aims to simplify this process by providing a universal system that any creator can use, without the need for lengthy negotiations.
The project’s success depends heavily on AI companies adopting the standard. Critics note that AI developers have often ignored robots.txt instructions, and without cooperation, it remains difficult to track usage and enforce fees such as those tied to inferences. Leeds acknowledged the challenge but argued that collective action increases pressure on AI firms. “Our job is to go out and get a big group of people to say it’s in your interest, both efficiently, because you can negotiate with everybody at once, and legally, because if you don’t, you’re violating everybody at once,” he said.
Enforcing compliance and prospects
Unlike some existing solutions, the RSL Standard cannot block AI crawlers on its own. To strengthen enforcement, the RSL Collective is working with Fastly, a content delivery network. Fastly will act as a gatekeeper by checking whether bots have agreed to licensing terms before allowing them access. “Fastly is the bouncer at the door to the club, and they won’t let people in unless they have the right ID,” Leeds explained. “RSL is issuing the IDs.”
Leeds also believes the system can be legally enforced. He compared it to established rights organisations such as ASCAP, which manage licensing fees in the music industry. While copyright laws in music are well defined, the use of online content for AI training remains in a legal grey area. Several lawsuits are ongoing, with Reddit, Getty Images, and several publishers challenging how AI models have used their material.
In a joint statement, Leeds and Walther said: “There has always been a question of whether bots have agreed to terms that they don’t see. RSL changes that fundamentally, putting crawlers on notice of what the terms are before they access a site.”
The RSL Collective is free for publishers and creators to join. Other companies, including O’Reilly, wikiHow, and Ziff Davis, owner of IGN, have also signed up in support. Leeds said the aim was not to reinvent the wheel but to provide missing infrastructure. “What we’re doing is not reinventing wheels or inventing wheels — we’re just bringing them to a place that they haven’t existed before,” he said.
The future of the standard will depend on whether AI firms agree to cooperate. But for publishers, RSL offers a potential framework to ensure fair compensation in the fast-changing AI landscape.
