Математика RNN: почему «детские» вопросы ставят инженеров в тупик

We're accustomed to perceiving neural networks as a black box that simply works if you feed it enough data and computational power. But dig a little deeper beyond the level of PyTorch or TensorFlow libraries, and it turns out that the foundation upon which modern AI rests is held up by things we often take on faith. Recurrent neural networks (RNNs) may seem like a relic of the past against the backdrop of transformers' dominance, but it is precisely in their structure that those very mathematical principles are embedded—principles without which it's impossible to comprehend the evolution of deep learning.

Once, RNNs taught machines to work with sequences, and understanding their "inner workings" is not merely an academic exercise, but a way to understand why modern models became what they are.

At the heart of any RNN lies the idea of transmitting state from past to future. Mathematically, this looks elegant until you start computing derivatives. Most textbooks helpfully offer us ready-made formulas, but rarely explain how exactly differentiation of a vector with respect to a matrix occurs.

For many engineers, this becomes a moment of truth: it turns out that the familiar rules from a high school calculus course work differently here, transforming into cumbersome operations with Jacobians. We often fear asking "childish" questions, such as why during backpropagation, gradients are added rather than multiplied at certain nodes in the graph. The answers to these questions lie in the very nature of the chain rule and how information flows through the layers of neurons.

The context of RNNs' emergence is closely tied to attempts to mimic human memory. However, in practice, researchers quickly encountered the problem of vanishing and exploding gradients. This is not merely a technical bug, but a direct consequence of the mathematical structure of recursion. When you multiply a matrix by itself dozens or hundreds of times during backpropagation through time (BPTT), any deviation of eigenvalues from unity leads either to signal annihilation or its infinite growth. It was precisely this mathematical impasse that forced the industry to seek alternatives, which first led to the creation of LSTM and GRU with their complex "gate" systems, and then to attention mechanisms, which formed the foundation of GPT architecture.

An analysis of the simplest RNN, like the one once popularized by Andrej Karpathy, exposes an industry irony: we build colossal systems on principles that still provoke debate over implementation details. For example, the question of how exactly to initialize weights to avoid learning collapse in the first few seconds remains more of an art than a strict science. We use heuristics that work, but we can't always explain "why" at the level of first principles. It's reminiscent of childish curiosity, when a child takes apart a toy to understand what's inside and discovers parts whose purpose not even adults understand.

Analysis of these fundamentals forces us to look differently at the current neural network boom. Understanding how difficult it was to make RNNs remember even a dozen words in a sentence, you truly begin to appreciate the engineering genius behind modern context windows spanning millions of tokens. However, old problems haven't gone anywhere—they've simply camouflaged themselves. Questions of computational efficiency and gradient stability remain relevant even for giant H100 clusters. Returning to the roots and examining "childish" questions about differentiation and error propagation allows you to shed senior-developer arrogance and see in the code not just a .backward() function call, but a complex and fragile dance of numbers.

The bottom line: fundamental understanding of RNN mathematics proves that there is no magic in AI—only long chains of derivatives that sometimes behave unpredictably due to our love of simplifications.