Hello, this is a news release I have compiled for you regarding the impact of AI crawlers on the internet. The content is concise and highlights key points.
According to the latest report from cloud service company Fastly, artificial intelligence (AI) crawlers are placing a heavy burden on the open web, as significant increases in website traffic are not coming from human visitors, but rather from an ever-growing number of robotic programs, especially crawlers and scraping programs from large AI companies.
The report states that among all AI robot traffic accessing websites, AI crawlers account for as much as 80%, while the remaining 20% is contributed by AI scraping programs. These scraping programs are triggered on demand when users request information after a specific date for training data from large language models (LLMs), causing explosive growth in traffic within a short period of time.
In terms of AI crawler traffic, the report reveals the proportion held by major technology companies: Meta has the highest share of AI crawler traffic at 52%, followed by Google and OpenAI, which account for 23% and 20% respectively; Anthropic and Perplexity AI have smaller shares, at 3.76% and 1.12% respectively.
Compared to AI crawlers, AI scraping programs generate even more volatile traffic. Fastly monitored that one scraping program can generate over 39,000 requests per minute. Notably, OpenAI dominates the AI scraping program traffic with an astonishing 98% share. This indicates that when users request real-time information from OpenAI's large models, it triggers a large number of immediate scraping activities, putting sudden immense pressure on website servers.
