The AI Giants: Nurturing or Exploiting the Web?
Are AI companies being generous contributors or greedy takers? Recent data from Cloudflare reveals a concerning trend: Anthropic and OpenAI are crawling the web extensively but sending very few referrals back. This behavior raises questions about the ethics of AI companies and their impact on the web ecosystem.
Here's the deal: Tech giants invest heavily in data centers and GPUs, but they often sidestep the crucial topic of data acquisition. They prefer dispatching bots to scrape websites for valuable human data rather than paying for it. This practice disrupts the original web bargain, where sites allowed their data to be used freely in exchange for referrals and revenue through ads and subscriptions.
But here's where it gets controversial: In the era of generative AI, chatbots and answer engines provide direct responses, reducing the need for users to visit the original sources of information. This shift has led to a breakdown in the web's grand bargain.
Cloudflare, which supports a significant portion of the world's websites, started monitoring this behavior in 2025. They track the crawl requests from Big Tech bots and the referrals sent to sites, creating a crawl-to-refer ratio. This ratio reveals how much these companies take from the web versus what they contribute.
For instance, a crawl-to-refer ratio of 100:1 means a company's bots crawl sites 100 times for each referral they provide. And guess what? Anthropic and OpenAI's ratios have worsened, indicating they are taking more and giving less.
This aligns with previous reports from late 2024, which highlighted how these companies' bots were causing traffic costs to skyrocket for some websites. One web developer witnessed a client's cloud-computing expenses double due to this AI bot swarm.
And this is the part most people miss: Not only are these AI companies taking more, but they are also leaving some site owners with higher costs. When asked about this, Anthropic remained silent, while OpenAI declined to comment.
A quick note: The crawl-to-refer ratios exclude native app activity, which might lower the ratios if included. Google's relatively low ratio is likely due to its traditional search engine, but even they are integrating AI-generated answers into their search results.
So, are AI companies being ethical in the AI era? The data invites us to reflect on this question. As we continue to monitor Cloudflare's data, we'll see if these companies change their ways. Stay tuned for more updates, and feel free to share your thoughts on this controversial topic in the comments.
