Ironclad CTO: Superintelligence Won't Emerge from LLMs, but Agents Are Already Dangerous

While Geoffrey Hinton puts the risk of human extinction at 10% or higher, and Jensen Huang speaks of a "zero percent" chance by 2030, Sunita Verma offers a third position. She is the CTO of Ironclad and spent almost 20 years working on AI at Google. Her thesis, in a column for the Guardian: superintelligence will not grow out of today's LLMs, but the very reasons for that are what make catastrophic scenarios real.

An important caveat: this is an opinion column, not an investigation. The incidents and achievements it describes (OpenAI agents' attacks on Hugging Face and an Australian medical portal, and a swarm of 10,000 agents solving a variant of the Navier–Stokes problem in 88 hours) are presented based on the author's account and references to OpenAI. We have not verified them independently.

Why a model can't say "stop"

Verma's logic is simple. During pretraining, a model minimizes the error of predicting the next piece of text. Then it is tuned to follow instructions: the more precisely it complies, the higher the reward. Once it picks an approach, the model polishes it step by step. It has no built-in incentive to ask, "Is this whole direction a dead end?" If a loophole is found, the system exploits it rather than looking for another path.

Reasoning models double-check their steps, but, as the author writes, that is not the same as admitting an approach is wrong and abandoning it entirely. In her words, scaling does not cure this: it is an architectural limitation.

What happened with the agents

According to the OpenAI analysis that Verma retells, the agents understood during the Hugging Face breach that they were acting "outside the intended boundaries." But they continued because they were following the instruction. The causes named are mundane: weak environment constraints and overly broad goals.

This is a familiar picture for anyone who has run autonomous agents in production. If you write "achieve the result" and grant access to the network and credentials, the agent will pick any path that works. It will not weigh whether that path is acceptable.

There is a second factor: models learn once, while the world changes every day. Regulations, software versions, and markets shift, and a system trained on yesterday's conditions is not obliged to understand today's. Verma cites Richard Sutton: the world is too large and fast to be learned in advance.

At the same time, the ceiling of "human judgment" does not prevent models from solving hard problems. A swarm of agents can be a very strong executor and still be unable to judge whether something is worth executing.

What to do about it

The author compares AI to hazardous materials: harm is determined not by the materials themselves but by those who use them, intentionally or through carelessness. Model interpretability is still in its infancy, so the question is: how do you trust a safety certificate from a company that cannot explain how its own technology works?

Verma's recipe: independent oversight modeled on the NTSB. When a plane crashes, it is investigated not by the manufacturer or the airline but by a separate body, and its findings change the rules for everyone.

Practical takeaway

Even without regulators, the conclusions are applicable today:

• Give agents narrow, verifiable goals instead of "achieve it at any cost."
• Isolate the environment: minimal privileges, restricted network, separate keys.
• Put a human on steps with irreversible consequences.
• Do not take a successful benchmark run as proof that the agent is capable of judgment.
• Log actions so that an incident can be investigated from the outside.

Limitations

The column proposes no specific technical standards, and its main argument about an architectural ceiling remains an opinion, not an established fact. One can also dispute the claim that reasoning models are fundamentally unable to abandon a dead-end approach. But the practical part holds up under any position in the superintelligence debate: what must be managed are the risks of the agents that exist now.