Виновата не модель, а архитектура: разбор 3 громких ИИ-инцидентов (Deloitte, ИИ-камеры, агент)

An engineer who builds production systems on large language models (LLMs) analyzed three high-profile AI incidents and concluded: in all three, the model itself is not to blame — the architecture missing around it is. The analysis was published on Habr.

What connects the three incidents

What connects the three incidents is not the type of error, but its source: the code around the LLM was written by people who know how to craft prompts but don't know how to design architecture. The holes ended up in three completely different parts of the system.

• Deloitte Australia refunded the government money for a report in which AI fabricated sources — the links were never checked by anyone
• In Minnesota, a network of AI cameras tracked a journalist as a car thief for several days, and four police cars blocked him in a parking lot
• The cause of the Minnesota incident was a data typo made two thousand miles from the journalist
• Code generated by the model canceled all customer subscriptions overnight for an indie SaaS and left the business with $38 in monthly revenue
• The author's overall diagnosis — an architecture failure, not a language-model failure

Three different holes in the architecture

The holes broke through at three different levels of the system. At Deloitte Australia, sources weren't checked: the model generated plausible but nonexistent links, and the report went to the client — the government — without fact validation. The result was a public refund of budget money.

In the Minnesota case, the data boundary failed. A network of AI license-plate recognition cameras held a journalist in "car thief" status for several days because of a single typo entered two thousand miles away from him. No one had built in a check or a way to dispute a false positive — which is why four police cars showed up at the man in the parking lot.

The third case is about an autonomous agent's permissions. An indie developer gave model-generated code far too much authority: overnight it canceled every customer's subscription. In the morning the founder discovered that only $38 was left of the monthly revenue. The autonomous agent was never given limits on destructive actions.

Why the model is not at fault here

In none of the three cases did the LLM itself break — what broke was what surrounds it. According to the author's analysis, the problem is systemic: engineers have learned to get the answer they need from the model, but they don't build checks, data boundaries, and permission limits around it.

"The systems around the LLM were built by people who know how to write prompts but don't know how to design architecture," the engineer's analysis on

Habr states.

All three holes — source validation, input data control, agent permissions — are classic engineering design tasks, not model fine-tuning tasks.

What this means

An LLM in a product is not a "smart feature" — it's a component that needs the same engineering safeguards as any critical system: output validation, input control, permission limits. Without them, it's not the model that fails — the product fails, and the price is quite concrete: from a budget refund to $38 in an account.

Frequently asked questions

Who is to blame for these AI incidents?

According to the author's conclusion — not the language model, but the architecture around it. All three systems were built by people who know how to write prompts but don't design checks, data boundaries, and permissions for autonomous agents.

How much did the SaaS founder lose?

Code generated by the model canceled all customer subscriptions overnight, and only $38 was left of the business's monthly revenue. The cause was that the autonomous agent's destructive actions were never restricted.

What happened with Deloitte Australia?

Deloitte Australia refunded the government money for a report in which AI fabricated sources. The links, generated by the model, were never checked by anyone before the work was delivered to the client.