Cloudflare's new marketplace enables websites to charge AI bots for scraping

Cloudflare said Monday that it plans to launch a marketplace next year where website owners can sell AI model providers to access content on their sites. The marketplace is the latest step in Cloudflare CEO Matthew Prince's broader plan to give publishers more control over when and how their AI bots scrape their websites. "If we don't compensate creators in some way, they're going to stop creating, and that's something we need to address," Prince told .

To achieve this, Cloudflare on Monday launched a free monitoring tool for customers called AI Audit. Website owners receive a dashboard that displays analytics on why, when, and how often AI models crawl their site for information, and Cloudflare enables customers to block AI bots on their site with a single click. Website owners can use AI Audit to block all web scrapers or allow specific ones if they have suggestions or if they think scraping would be profitable.

An AI Audit demo provided and showed how website owners can use the tool to see how an AI model would take down their site. The Cloudflare tool can see where each scraper comes from, which visits your site and offers selective windows to see how many times the disassembly of Openai, Meta, Amazon and other suppliers of artificial intelligence models visit your site .

Cloudflare tries to solve the problem approaching in the artificial intelligence industry: how will small publishers survive the AI ​​era if people go to Chatgpt instead of their website? Currently, AI model providers crawl thousands of small websites for information to support LLM. While some large publishers have content license agreements with OpenAI, most websites receive nothing and continue to feed their content to popular AI models every day. This could destroy the business models of many websites and cut off much-needed traffic. Earlier this summer, AI startup Perplexity was accused of using the Robot Exclusion Protocol to scrape websites that had purposely indicated they didn't want to be crawled. Shortly after, Cloudflare released a button that allowed customers to block all AI bots with one click. "It started with complaints we were hearing about people who felt their content was being stolen," Prince said.

Some website owners told Business Insider that the AI ​​bots were probing their websites so actively that it felt like a DDoS attack had paralyzed their servers. Getting your website hacked can not only be frustrating, but it can literally increase your cloud bill and impact your service.

But what if you want to block Perplexity bots but not OpenAI? Prince told TechCrunch that Cloudflare customers are asking for tools that let them choose which AI models have access to their sites. Cloudflare’s new tools, launched today, will let customers block some AI crawlers while allowing others through. Prince said that even major publishers that have licensing agreements with OpenAI, such as TIME, Condé Nast, and The Atlantic, have relatively little information about how much data ChatGPT collects from their websites. Many of them will have to accept what OpenAI has to say, but their response will determine whether publishers sign the appropriate licensing agreements. But Cloudflare's Marketplace, due to launch next year, aims to let smaller publishers strike deals with AI model providers.

"Let's give you the opportunity to do something that's only been done by Reddit, Quora, and the big publishers of the world," Prince said. "What if we could essentially set a price for the access and transmission of content that gets loaded into these systems?"

It's a bold idea, but Cloudflare hasn't shared a full idea of ​​what its marketplace would look like. Prince said websites could potentially charge AI model providers based on how much they extract data from individual websites, but it's unclear how much they would actually pay. Further, he says websites could charge a monetary price to be scraped, or simply ask AI labs to give them credit. The details are fuzzy. While AI companies may not be initially excited about paying for content that's currently available for free, the Cloudflare CEO said he thinks it's ultimately a good thing for the AI ​​ecosystem. Prince said the current situation, where some AI companies don't pay for content at all, isn't sustainable.