A Physical Transformer Built Entirely From Transistors Learns In-Context Learning in Simulation

A group of researchers has shown in simulation a physical system that learns without digital arithmetic and still handles in-context learning, a capability usually associated with transformers. The circuit consists entirely of transistors and trainable resistors. MOSFETs in the subthreshold regime perform the multiplication needed for dot-product attention. Each parameter of the Lion optimizer is stored in a single capacitor. The work appeared on arXiv in the cs.LG section. It is a preprint, and there is no hardware in silicon yet.

Why physical learning has been stuck at multilayer perceptrons

Equilibrium propagation and coupled learning belong to physical learning. The system is nudged from a free state toward a target state, and a local rule updates the parameters based on the difference between the two states. This requires information to flow through the physical system in both directions, from input to output and back. In simple circuits, that happens on its own. Such systems remain within the expressive class of multilayer perceptrons. They can approximate any function, but they handle poorly the associative tasks that transformers solve easily.

The cause is nonlinearity. The attention mechanism has to multiply quantities, and circuits that do so break the backward flow: the signal no longer returns in the way needed to compute the gradient. The authors went straight at this problem.

The dyadic circuit and the MOSFET exponential

They restored the backward flow explicitly. They added extra elements to the dyadic circuit that are responsible for moving the signal backward. The attention mechanism is built on top of this. It uses the exponential current law of a subthreshold MOSFET: the current grows exponentially with the gate voltage, and the product of exponentials equals the exponential of the sum. This turns multiplication into an operation the transistor performs by itself. The same MOSFET block is used for softmax.

In the simulation, the backward components are realistic rather than ideal. They pass only an approximate gradient. Training still converges on a simple in-context learning task.

Key parameters of the work:

• Platform: a fully transistor-based circuit, verified in simulation.
• Attention: the exponential current law of a subthreshold MOSFET.
• Backward flow: extra elements in the dyadic circuit, with an approximate gradient.
• Loss function: cross-entropy only; training failed with mean squared error.
• Nudge signals: a new floating element based on the same MOSFET subcircuit as softmax.
• Trainable parameters: resistors.
• Optimizer: Lion, with one capacitor per parameter serving as local memory of past gradient estimates.
• Task: simple in-context learning.

The finding about the loss function matters to engineers. In such a system, the choice of loss determines whether training works at all, and that constrains the design space.

What to make of it and what to expect

For ML practitioners there is no direct benefit right now: there is nothing to deploy, and the task in the paper is deliberately simple. The main value lies elsewhere. The authors showed that analog in-place learning can move beyond perceptrons and reach the architecture that underpins today's LLMs. Lion, which needs only a capacitor per parameter, suggests that the question of local memory for the optimizer has also been thought through at the circuit level.

The main questions remain open. First, whether the circuit can make the journey from simulation to silicon. Subthreshold transistors are known for sensitivity to temperature and parameter variation, and the abstract does not address these effects. Second, how the system will behave on tasks harder than simple in-context learning and with more parameters. Third, what the real energy figures will be: the abstract gives no such numbers, and without them a comparison with GPUs is impossible.

Three things are worth watching: the first physical prototype, a broader set of tasks, and measurements of energy per operation. If these steps are completed, the approach could earn a niche where low power consumption and on-device training matter.