AI Agent Security in Production: A Practical Red Teaming Guide

An agent is not a chatbot. It is a system with access to tools, services and corporate data. A model error in an isolated chat is an embarrassment. An error by an agent with access to email and documents is a potential data leak, or a reputational or financial incident.

How agent Red Teaming differs

LLM Red Teaming focuses on the language model itself: testing prompt injection, jailbreaks and hallucinations. If the model answers incorrectly, that is a local problem.

Red Teaming an agent is a different matter entirely. Here we test the whole stack: the model, tools, external APIs, integrations with corporate systems, and request-routing logic. An agent can answer questions correctly yet pick the wrong tool, pass the wrong parameters, or forget to check access rights. And suddenly the agent performs an action it should never have performed. One error in this chain is an incident.

Doubletapp developed a Red Teaming methodology that covers both levels: vulnerabilities in the model itself, plus vulnerabilities in its integration with the outside world.

Promptfoo: from theory to practice

Promptfoo is a framework for automating Red Teaming. You define test scenarios (attack scenarios), a set of dangerous prompts and rules for evaluating results. The tool runs these tests against your agent and generates a report showing which attacks succeeded.

The basic workflow is simple:

• describe the behavior you want to protect;
• write test scenarios, which are attempts to make the agent violate that behavior;
• run Promptfoo, which automatically runs all the tests;
• read the report and work out where the holes are;
• close the vulnerability and repeat.

The tool integrates with OpenAI, Anthropic, Claude and other models. All logs are transparent, detailed and easy to analyze.

Which vulnerabilities to look for

In practice, Doubletapp has run into recurring classes of problems:

• Incorrect tool authorization: the agent picks the right tool but does not check whether the user has permission for that operation
• Parameter confusion: the agent passes user_id instead of admin_id because of unclear naming in the API specification
• Chained attacks: one small error plus another small error add up to a complete system bypass
• Social engineering through the model: an attacker convinces the agent that it is authorized when it actually is not
• Context leakage through logs: the agent logs sensitive data that another user later sees

"This is the first step in a process, not the final product" is roughly what people say about any Red Teaming.

The first round of testing will expose holes that then have to be closed wave after wave.

What this means

Red Teaming is moving out of the lab and into operational reality. If you have already launched an agent in production, you need a system that continuously looks for vulnerabilities. Promptfoo is one tool you can set up right now and use on your own stack.

Businesses are starting to demand not just functionality but proof of security. That is the right demand, because the stakes are high.