It's Not the Model, It's the Architecture: Breaking Down 3 High-Profile AI Failures (Deloitte, AI Cameras, an Agent)

An engineer who designs production systems built on large language models (LLMs) has analyzed three high-profile AI failures and concluded that in all three, the fault lay not with the model itself but with the missing architecture around it. The analysis was published on Habr.

What the three failures have in common

What links the three incidents is not the type of error but its source: the code around the LLM was written by people who know how to write prompts but not how to design architecture. The holes, however, turned up in three completely different places in the system.

• Deloitte Australia refunded the government for a report in which AI invented sources; nobody checked the links
• In Minnesota, a network of AI cameras tracked a journalist as a carjacker for several days, and four police cars boxed him in at a parking lot
• The cause of the Minnesota incident was a typo in the data, made two thousand miles from the journalist
• Model-generated code canceled all the customer subscriptions of an indie SaaS overnight and left the business with $38 in monthly revenue
• The author's overall diagnosis: a failure of architecture, not of the language model

Three different holes in the architecture

The failures broke through at three different levels of the system. At Deloitte Australia, sources went unverified: the model generated plausible but nonexistent references, and the report went to the client, the government, without any fact validation. The result was a public refund of taxpayer money.

In the Minnesota case, the data boundary failed. A network of AI license-plate-recognition cameras kept a journalist flagged as a carjacker for several days because of a single typo entered two thousand miles away from him. Nobody built in a check or a way to contest a false match, so four police cars showed up at the man in the parking lot.

The third case is about the permissions of an autonomous agent. An indie developer gave model-generated code too much authority, and overnight it canceled every customer's subscription. In the morning, the founder found that $38 of the monthly revenue was left. The autonomous agent had no limits on destructive actions.

Why the model isn't to blame

In none of the three cases did the LLM itself break; what broke was everything around it. According to the author's analysis, the problem is systemic: engineers have learned to get the answer they want out of a model, but they don't build checks, data boundaries and permission limits around it.

"The systems around LLMs were built by people who know how to write prompts but not how to design architecture," the engineer writes in the

Habr analysis.

All three holes (source validation, input data control, agent permissions) are classic design problems, not model fine-tuning problems.

What this means

An LLM in a product is not a "smart feature" but a component that needs the same engineering safeguards as any critical system: output validation, input control, and limited permissions. Without them, it won't be the model that fails but the product, and the price can be very concrete: from refunding taxpayer money to $38 in the account.

FAQ

Who is to blame for these AI failures?

According to the author of the analysis, not the language model but the architecture around it. All three systems were built by people who know how to write prompts but don't design checks, data boundaries and permissions for autonomous agents.

How much did the SaaS founder lose?

The model-generated code canceled all customer subscriptions overnight, leaving the business with $38 of its monthly revenue. The cause: the autonomous agent had no limits on destructive actions.

What happened with Deloitte Australia?

Deloitte Australia refunded the government for a report in which AI invented sources. Nobody checked the model-generated links before the work was delivered to the client.