Cloudflare recently announced the launch of a groundbreaking marketplace - "Pay-per-Scrape" - aimed at reshaping the relationship between website owners and artificial intelligence (AI) companies. This new marketplace will help content publishers better control their content while providing AI companies with a fair way to scrape content.
Over the past year, Cloudflare has introduced several tools to address the surge in AI crawlers, including a one-click solution to block all AI crawlers and a dashboard to monitor AI crawler activity. According to Cloudflare's CEO Matthew Prince, these tools laid the foundation for the new market.
In the "Pay-per-Scrape" marketplace, website owners can decide whether to allow AI crawlers to scrape their websites at specific rates, choose to offer it for free, or even completely block crawler access. This model provides more options for websites, allowing them to control the rights to their content.
The launch of this market comes at a time when the news publishing industry is facing significant challenges, especially against the backdrop of decreasing traffic from Google Search and the growing popularity of AI chatbots. While some major newspapers have signed content licensing agreements with tech companies, the majority of small and medium-sized publishers still do not receive fair revenue distribution. Cloudflare hopes that this innovative business model will provide more publishers with a sustainable source of income.
At the same time, Cloudflare also announced that all newly created websites will default to blocking AI crawlers, and website owners must individually grant permissions to certain crawlers. This measure has received support from several large publishers, including Condé Nast, Time Magazine, and Associated Press, who all hope to achieve a "permission-based crawling" model.
However, the challenges brought by the AI era to publishers cannot be ignored. Recent data shows that AI crawlers scrape content far more frequently than Google crawlers. For example, OpenAI's crawler needs to scrape content 1,700 times to bring recommended traffic to publishers, while Anthropic requires 73,000 times. This poses a serious challenge to publishers relying on user visits to their websites as a business model.
Cloudflare also stated that the true potential of this market lies in the widespread application of future "intelligent agents." Imagine users obtaining the latest research results or legal information through intelligent agent programs, and these agent programs can pay for high-quality content on demand.
AI companies and publishers participating in this experiment need to register an account on Cloudflare and set prices and transactions through the platform. Although cryptocurrency is not currently used, Cloudflare's CEO hinted that stablecoins may be explored for transactions in the future.
Cloudflare's "Pay-per-Scrape" marketplace offers publishers a new path to profit, although it still faces many challenges, its potential should not be underestimated.