The Wall for LLMs: Why the Skeptics Picked the Wrong Door Again

Every six months, the AI industry enters a season of "great gloom." First we were told GPT-3 was the limit and there was nowhere left to grow. Then we were assured that the next step required trillions of tokens that simply don't exist on the internet. Now the new trend: the data is gone, the transformer architecture has run its course, time to go home. It sounds weighty and even logical, if you look at the world through a keyhole. But if you've followed the game for longer than a single hype cycle, you understand: we haven't hit a wall. We've reached the end of one straight road and are turning onto an expressway.

The skeptics' problem is that they think in one dimension. To them, progress means pouring more text into a model and getting more intelligence out. True, text data for classic supervised training really is running out. The entire internet has already been "digested" by neural networks. But training on data is just one of six axes along which progress moves. While some mourn the empty libraries, engineers at OpenAI, Google and Anthropic are busy turning the other five levers, which for some reason get forgotten in public debates.

The first and arguably most important lever today is compute at inference time, or inference-time compute. Look at the o1 family of models. They don't just spit out an answer; they "think" before writing the first letter. This changes the paradigm: you no longer have to make a model ten times bigger to make it smarter. You can make it reason longer about the problem. It's like life: a smart person is not the one who has read the most books, but the one who knows how to analyze information deeply. We are moving from quantity of reading to quality of understanding.

The second axis is algorithmic efficiency. Remember how everyone complained about how power-hungry transformers are? Now architectures like Mamba and Mixture of Experts (MoE) are stepping onto the stage. They deliver the same power at far lower resource cost. We are learning to build better engines, not just enlarging the fuel tank. The third axis joins them: multimodality. Models are ceasing to be mere "text readers." They are starting to see, hear and understand the physical world. When AI learns from video and audio, the notion that "text has run out" loses all meaning. The world is an endless stream of data that we have only begun to tap.

The fourth and fifth factors are tool use and self-improvement through self-play. Remember how AlphaGo beat the world champion at Go. It didn't learn only from human games; it played against itself millions of times. Now this approach is coming to LLMs. Models are starting to generate synthetic data, check it for logic and learn from their own mistakes. If an AI can create tasks and solve them itself, it no longer needs humans as its only source of knowledge. This closes the learning loop and makes it practically endless.

We are at a point where the old progress metrics, parameter count and dataset size, are no longer the main ones. An era of architectural flexibility and intellectual depth is arriving. Those who shout today about "the exhaustion of technology" simply haven't noticed that the rules of the game have changed. We haven't reached the ceiling; we have just finished the foundation and are starting to build the floors. And judging by how fast agents and tools are being adopted, those floors will rise much faster than anyone expected.

The key point: Forget the "data shortage." The real battle now is over who teaches a model to think longer and more efficiently, not who feeds it more terabytes from Reddit.