AI bots capture traffic: why the internet is turning into a closed party

Remember when the biggest threat to your website was just a regular DDoS attack or a flood of spam comments? Those days now seem like a leisurely walk in the park. Today the internet is experiencing a quiet but massive transformation: it's ceasing to be a space for people and turning into a vast buffet for artificial intelligence. Fresh data shows that AI bots have become not just noticeable, but a critical source of traffic, and this is forcing publishers worldwide to build actual barricades. We're entering an era where every byte of your text is fuel for someone else's neural network, and nobody will pay you for it.

Context here matters more than the numbers themselves. For decades, there was an unspoken pact between content creators and search giants like Google. You allowed the search engine to index your pages, and in return it sent you users.

It was a fair deal: data in exchange for traffic, which later turned into ad views or subscriptions. But AI agents and new-generation search bots have shredded this pact to pieces. When a Perplexity or OpenAI bot visits your resource, it has no intention of bringing readers to you.

It wants to extract the essence of your text, package it into a neat answer, and serve it to the user in its own interface. The user no longer needs to follow a link. The publisher is left with zero in the "revenue" column and a massive bill for server resources consumed by this bot.

The situation has become so acute that the largest media conglomerates have started mass-blocking the User-Agent of popular models. If robots.txt used to be considered a gentleman's agreement that everyone followed, today it's more like a leaky fence.

Some AI companies have already learned to bypass these restrictions by masquerading their bots as regular users. In response, publishers are implementing more aggressive protection methods: from hard paywalls that even the smartest parsers can't break through, to advanced traffic filtering systems from Cloudflare and other cybersecurity giants. We're witnessing the beginning of a genuine arms race, where on one side stand billions in investments for model training, and on the other—the desperate desire of media outlets not to become free fuel for algorithms.

What does this mean for us? First of all, the internet will become less accessible. To protect themselves from bots, websites will require authentication even to read short posts. We already see how Reddit and Twitter (X) have closed their doors, turning into closed ecosystems. This creates a dangerous precedent: if all quality content disappears behind high walls, future AI models will be trained on garbage that remains in the public domain. A paradox of cannibalization emerges: AI devours the very environment it needs to develop. Without an influx of fresh human texts, which are now actively being blocked, the quality of neural network responses will inevitably start degrading in a couple of years.

Moreover, the very nature of the web is changing. We're used to an "open internet," but now it's rapidly fragmenting. Publishers no longer trust free traffic and are moving toward direct licensing deals with tech giants. Those who can't reach agreements with OpenAI or Apple for payments for their content risk simply disappearing, as their presence in search will become meaningless. This is a battle for survival in which old SEO rules no longer work, and new ones haven't been written yet. While we debate whether AI will become smarter than humans, it's already doing something more mundane—stripping content of its economic foundation.

The bottom line: the era of a "free and open" web is officially ending, and the time of closed platforms and paid access is beginning. Will AI companies be able to find a balance between their hunger for data and preserving the ecosystem that generates that data, or will they ultimately burn out the digital soil beneath them?