Implicit CoT: How Neural Networks Learned to Think Without Saying a Word

When OpenAI introduced the o1 model, the world came to grips with the concept of a "chain of thought" (Chain of Thought) for the first time. We were used to neural networks producing an answer instantly, but o1 made us wait while it "muttered" to itself, weighing options. It looked like magic, but the magic had a high price: waiting time and a huge number of tokens spent on internal reasoning that the user might never even see.

This is so-called explicit reasoning (Explicit CoT), which became a temporary crutch for models trying to imitate human logic. But new research on implicit reasoning (Implicit CoT) promises to rid us of this redundancy. The researchers asked whether a model really has to "speak out" every logical step to reach the right conclusion.

It turned out that it does not. With specialized training and knowledge distillation, a neural network can be taught to hide these intermediate stages inside its hidden states. This fundamentally changes the paradigm: instead of spending compute on generating text that nobody reads, the model directs it toward producing the correct answer directly.

To grasp the scale of the change, picture the difference between a schoolchild who solves an equation by writing down every step and a professor who sees the solution instantly. OpenAI o1 is the diligent schoolchild. Implicit CoT is an attempt to grow that schoolchild into a professor.

Moving reasoning out of text output and into internal computation makes it possible to reach the same accuracy on math and logic tasks while saving enormous amounts of resources. For the industry, this means future models will be not only smarter but also significantly faster and cheaper to run. The shift also addresses the problem of context "pollution."

When a model generates thousands of reasoning tokens, it can get tangled up in its own words. Hidden reasoning avoids that noise. The researchers used reinforcement learning methods to force the model to "compress" its thoughts.

As a result, the network learns to operate on higher-order abstractions without breaking them into primitive textual explanations. In effect, this is a step toward what Daniel Kahneman called "System 1": fast, automatic, intuitive thinking that nonetheless rests on deep logical groundwork. For developers and businesses, it is a signal that the race for parameter count may finally give way to a race for architectural elegance.

If we used to think that solving complex tasks required giant context windows and endless compute at inference time, it is now becoming clear that efficiency lies in a model's ability to internalize its knowledge. We are on the threshold of a new generation of LLMs with the depth of o1 and the speed of GPT-4o. This is not just optimization; it is the maturing of a technology that is finally learning to think to itself before speaking.

The takeaway: the era of "long thoughts" aloud was only a transitional stage. Can open models master Implicit CoT faster than OpenAI closes this gap in its commercial products?