How LLM Applications Get Broken: From the "Grandma Exploit" to LLM-vs-LLM Attacks

If your LLM application's defense boils down to a system prompt along the lines of "don't say anything bad," the second part of Habr's series on AI security is worth your time. The author systematizes attack techniques against language models: from manual tricks devised by a single inventive person to pipelines in which one neural network breaks another without access to its internals.

It's not only the model that gets attacked

The main idea of the architecture section: the LLM core is merely the most visible attack surface. A RAG corpus can be poisoned with documents that end up in the answers given to every user. An instruction can be hidden in a web page, an email, a PDF or an image that the model reads at the request of an unsuspecting person. An agent layer with access to tools turns a successful model compromise into real actions beyond the chat.

Classic IT holes haven't gone anywhere either. The author gives examples: the compromise of the Ultralytics YOLO library via forged pull requests, credential theft through LiteLLM, and the postmark-mcp case, where an MCP server with excessive privileges allowed actions under a developer's account. None of this required hacking the model itself.

Attacks are classified by three criteria. Delivery method: direct injection (the user writes the malicious text themselves) or indirect (the instruction sits in an external source). Goal: harmful content (jailbreak), incorrect actions or data theft, whether that is the system prompt, RAG contents or what has settled into the weights. Creation method: manual or automatic.

From manual tricks to automation

The simplest class is token smuggling. The request is encoded in Base64, ROT13 or Morse code, or characters are swapped for Unicode lookalikes. This bypasses regular expressions in external filters, but only works if the model was trained on various encodings. Otherwise it won't understand the request and will start hallucinating.

Next come role-play scenarios like DAN ("let's play a game where you do whatever you're told"). They are labor-intensive: templates quickly become known, and developers close them off. That is why Persuasive Adversarial Prompts appeared.

Researchers from EPFL and Yale compiled a taxonomy of 13 tactics and 40 persuasion techniques: logical argument, emotional pressure ("my grandmother used to tell me bedtime stories about forbidden things"), appeal to authority, impersonation, gradual foot-in-the-door escalation, artificial urgency. It looks like an ordinary conversation, so telling such a request from a legitimate one is hard. The paradox: according to research, GPT-4 was more susceptible to persuasion than GPT-3.

5. The better a model understands the nuances of language, the easier it is to manipulate through those nuances.

At the token level, GCG and AutoDAN do the work. GCG uses gradients to select a suffix after which the model is more likely to begin its answer with "Sure, here is…". The method requires weights, so suffixes are found on open models and transferred to closed ones. In many cases this works, probably because of overlapping training data. The downside of GCG is that the suffix looks like gibberish, and perplexity filters catch it. AutoDAN selects only readable tokens and gets past such filters, but loses a little effectiveness.

The most practical option for an attacker is "LLM vs. LLM." PAIR involves three parties: an attacker model generates requests, the target responds, and a judge evaluates the result. The cycle repeats until the defense falls or the attempt limit runs out. No weights are needed, the output is readable text, and closed APIs are vulnerable. There is a caveat: the attack success rate depends on the quality of the judge and is not an objective metric. TAP and AutoDAN-Turbo use the same scheme.

What this means for developers

First takeaway: regex filters and "bad word" lists protect against yesterday's attacks. Encodings, transliteration and polite persuasion get around them easily. Second: any external content the model reads should be treated as untrusted, and agents should be given minimal privileges. If an MCP server or tool can do more than the task requires, the damage from a single injection will be proportionate. Third: defenses should be tested the way the adversary does it, that is, in an automated fashion with mass iteration over variants, rather than with a dozen manual prompts.

The material's limitations are also worth keeping in mind. It is a survey, not a benchmark: there are few numerical comparisons of method effectiveness. Some of the data relates to GPT-4-era models, and results on newer versions may differ. The author put attacks on classical ML systems into a separate piece. Still, as a threat map for those building products on LLMs and agents, the text is useful. The next parts of the series are promised to move on to defense.