All news

Deploying AI agents: know your risks

Deploying AI agents in production comes with serious risks, and not only security ones. This article gives you a mental model to understand yours.

On this page

AI agents come with major risks: they are non-deterministic systems (the same input can produce different outputs) that plan and execute actions in production, based on instructions you do not fully control. This doesn't mean you should stop using them, only that you need to know your risks before you deploy them.

Which risks you face depends on what your agent can do and, in our experience, every agent falls into one of four stages, each inheriting the risks of the previous one and adding new ones:

Defining Risks

First things first, we need to define what we mean by risk: something that may happen and hurt your company. When people hear "AI agent risks", they usually think of security: jailbreaks, prompt injection and leaked data. That is part of the picture, but only a part: your agent can also give your users wrong answers, say something that ends up in the news or promise a refund you then have to honor. None of these needs an attacker, as they can all happen in a perfectly ordinary conversation with one of your users.

The Four Stages

Now that we know what a risk is, let's go back to the four stages in the picture above. Each one is defined by what your agent can do:

  1. Talk: your agent takes the user's query, adds a system prompt and asks an LLM to produce an answer
  2. Read: your agent also looks up information, such as your return policy from the help center or the results of a web search, and uses it to write the answer
  3. Act: your agent can also plan and carry out actions through tools, that is, functions that do something beyond producing text, such as updating a record or sending an email
  4. Collaborate: your agent also hands tasks to other agents and receives tasks from them, and can start working on its own when something happens, such as a new ticket being opened on your CRM

In the next sections, we'll go stage by stage and present their risks. Some of them may sound far-fetched, and you may be tempted to dismiss them with a yeah, but users won't ever do that. Before you do, make sure that's actually true, because once your agent handles thousands of conversations, someone eventually will. And when they do, your agent won't push back much: LLMs are trained to be helpful, often to the point of breaking your rules just to satisfy a user.

Talk

Talk is the simplest stage, and the one most AI agents start from: your agent receives a query, wraps it with a system prompt and returns whatever the LLM generates. With no external data and no tools, it is often considered harmless, but it isn't: every risk here lies in what your agent says.

Reputational damage, risk 1 of 4

Your AI agent speaks on behalf of your company, and your users will take whatever it says as your company saying it. This can range from badmouthing your own services or recommending a competitor to toxic answers and full-blown PR nightmares. It happened to DPD in 2024, when a frustrated customer got its chatbot to swear and write a poem about how terrible the company was (The Guardian).

If your agent promises something it shouldn't, like a refund, your company may have to honor it. This has already happened to Air Canada: its chatbot told a customer they could claim a bereavement refund retroactively, and when the company refused to pay, it went as far as arguing that the chatbot was a separate legal entity responsible for its own actions. The tribunal disagreed and held Air Canada responsible for what its chatbot said (BBC).

Every token your AI agent generates ends up on your bill, and a number of strategies exist to trick it into generating lots of useless ones. The simplest one is also surprisingly effective: just ask the model to repeat something N times. A single request won't break the bank, but script it over a few thousand conversations and you get an inflated bill. That's actually the good outcome: if you run on prepaid credits, or your provider has rate limits, the attacker burns through what your real users needed and causes an actual outage, which makes this a Denial-of-Service (DoS) attack under another name.

Your system prompt is part of every conversation, so assume any user can get your agent to repeat it, often just by asking in the right way. The problem is rarely the leak itself, but what ends up in the prompt: credentials, internal pricing rules or notes on specific customers have no business being there. And even when nothing in it is sensitive, a leaked prompt hands an attacker the exact rules they need to work around. Treating your system prompt as a secret only gives you a false sense of security: write it as if it were public.

1 / 4

You may have noticed that prompt injection is missing, even though it's usually the first thing that comes to mind when talking about AI agent risks. This is intentional: prompt injection is not a risk, but a technique an attacker uses to steer your agent away from safe behavior and make one of the risks above materialize. Those risks are what you actually care about.

Read

In Read, your agent also looks up information before answering, such as a page from your help center or the results of a web search. This changes more than it seems: in Talk, your agent's input came from two places only, the user's messages and the system prompt you maintain, while now there is a third, whatever your agent fetches. You don't fully control it either, and it can steer the answer by accident, like an outdated document, or by design, like instructions someone planted there.

In the latter case, you are looking at indirect prompt injection: the same technique we just discussed, except that the attacker doesn't need to talk to your agent at all, only to plant their instructions somewhere it will read, from a web page to your own internal documentation.

On top of Talk's risks, Read adds:

Information correctness, risk 1 of 2

Your agent could already make things up in Talk (the infamous hallucinations), but Read adds new ways to be wrong, on both ends: your sources can be outdated or contradict each other, and your agent can fetch the wrong page or misread the right one. Either way, the answer inherits the mistake with full confidence, and it costs you more than before: once your agent looks things up, you delegate more to it and your users trust its answers more.

Whatever your agent reads can end up in its answers, which becomes a major risk as soon as it reads resources meant to stay internal, such as internal documentation or customer records. If what lands in an answer is another user's personal data, you are looking at a full-blown privacy breach. At this stage, the leak mostly stays within the conversation, but not always: many chat interfaces automatically load the images an answer points to, so an attacker can get your agent to hide data in an image's web address, and loading the image delivers it straight to them. This is how EchoLeak worked against Microsoft 365 Copilot in 2025 (SOC Prime).

1 / 2

Act

Read widened what your agent listens to, while Act changes what it can do. This is where agency comes in: your agent stops simply answering and starts reasoning about the request, planning the steps to fulfill it and carrying them out through tools, from internal functions to external APIs. This is very powerful, but it's also where the stakes rise: the more access you give your agent, the more damage it can do when it goes wrong.

On top of Read's risks, Act adds:

Information leak, risk 1 of 3

With Read, a leak mostly stayed within the conversation: whatever your agent disclosed, it disclosed to the user in front of it. Tools remove that boundary, as an agent that can send emails or call APIs can move data anywhere those tools reach. This is why information leak appears twice in the picture: it's the same risk as in Read, but it can now reach far more people. This is roughly what happened with ForcedLeak, a vulnerability found in Salesforce's Agentforce in 2025 (Noma Security).

Every tool you give your agent is something it can get wrong, by mistake or because someone told it to. This makes the risk a matter of how you set your agent up: it should have only the tools it actually needs, each with permissions limited to its task, and it should wait for a human to confirm before any action that matters.

Many agents have memory, that is, a tool to save what they learn in a conversation and reuse it in later ones. This is what makes them feel personal and it is also how a conversation can outlive itself. Whatever your agent writes to memory, it will later read back, so a single conversation, malicious or simply mistaken, can poison how your agent behaves from then on, until someone notices and cleans it up.

1 / 3

Collaborate

Of course, since a single AI agent is already problematic, we humans decided to put several of them together and give them even more autonomy.

In Collaborate, a request is split among several specialized agents, each with its own instructions, sources, tools and permissions, which hand work to each other until the job is done. Your agents can also start working on their own when something happens, such as a new ticket arriving or a scheduled check. This is how you go from automating single tasks to automating entire workflows.

The problem is that your agents now talk to each other, and their communication is a new layer to defend: whatever one agent sends ends up in another's context, and the receiving agent can act on it, hallucinations or adversarial triggers included. And when something does go wrong, finding where is much harder.

On top of Act's risks, Collaborate adds:

Rogue agents, risk 1 of 4

An agent goes rogue when it uses the access you gave it in ways you never intended. If this reminds you of excessive agency from Act, the difference lies in where the problem is. With excessive agency, the problem is what your agent can do, and you fix it by giving it fewer tools and permissions. With a rogue agent, the problem is what it decides to do, so even a correctly set up agent can go rogue, be it because of an adversarial input, a hallucination or a bad plan. Replit's coding agent did exactly this in 2025: told not to change anything during a code freeze, it deleted a company's production database anyway (and then wrongly claimed it couldn't be restored) (Fortune).

With a single agent, a mistake stays where it happens. With many, one agent's output is another's input, so a single error, be it a hallucination, a poisoned document or a malicious instruction, gets passed on, built upon and often amplified at every step. By the time it surfaces, it has gone through several agents, each acting reasonably on what it received, and it has done far more damage than the original mistake ever could.

To get their work done, your agents (at least some of them) rely on components someone else built and maintains, such as third-party tools and MCP servers, software packages and, increasingly, other companies' agents. If one of these components gets compromised, your agent could be too, without the attacker ever touching your systems. It has already happened: in 2025, an unofficial copy of the MCP server of Postmark, an email provider, started quietly copying every email it sent to an attacker (Postmark blog).

With a single agent, access is easy to reason about: the agent can reach whatever you gave it, and nothing more. In Collaborate, this no longer holds, as access depends on how the work unfolds: for instance, a low-privilege agent can recruit a high-privilege one to do what it couldn't do itself. As a result, your agents together can reach far more than any of them was meant to, and since this only shows up at runtime, you won't catch it by reviewing their permissions one by one. Far-fetched? This is roughly what researchers showed in 2025 on ServiceNow's AI agents (AppOmni).

1 / 4

Conclusions

The four stages give you a quick way to know what you are getting into with any AI agent: look at what it can do, find its stage, and you already have a first list of what can go wrong.

That list is not exhaustive, especially for Collaborate: it is a wide stage, and we focused on the setups most companies are deploying today. As agents get more autonomous, writing and running their own code or working for days with nobody checking in, the risks we covered get sharper and new ones join them. If that's where you're heading, the OWASP Top 10 for Agentic Applications is a good next read.

Either way, knowing your risks is only the first step: next, you need to find out which of them actually affect your agent, and how to fix them. That's what we work on at Principled Intelligence, so if you'd like to find out where your agent stands, get in touch.

Keep reading
Guardrails

Introducing unsafe-eval: a framework for AI safety evaluation

When OpenAI, Anthropic, Google, or others release a new LLM, we're curious whether it makes sense to use it as a safety guardrail. Trying a few prompts out usually gives us a first impression; however, a proper comparison with other models takes more work. We built unsafe-eval to make those comparisons easier to run, and we're now sharing it.

11 min read
Guardrails

Introducing ScopeGuard-2

Sometimes your AI assistant should use a reply you've already written, or leave a request to someone on your team. Following feedback from our users, we built ScopeGuard-2 to recognize when each is needed. We're releasing it with more control over what each prediction returns, and new models that match frontier models on accuracy in our evaluations while responding much faster.

8 min read