Testing AI agent resilience to prompt injection – a 5-layer methodology

Updated: 10 September 2026

What does testing AI agent resilience to prompt injection involve? It involves establishing whether the agent will take an action contrary to the user’s intent after reading content an attacker planted. The critical difference from conventional security testing lies in the object of study. You are not testing a model’s response to a malicious prompt; you are testing the whole loop: content retrieval, decision, tool invocation and memory write. A single probe fired at a model endpoint will never see this class of vulnerability.

Canonical definition: Indirect prompt injection is an attack in which an instruction for the model is placed in content the agent processes while performing a task, rather than entered directly by the user in the conversation.

Testing agent resilience to prompt injection – in brief:

1. The baseline risk taxonomy is the OWASP Top 10 for Agentic Applications, published on 9 Dec 2025 by the OWASP GenAI Security Project with identifiers ASI01-ASI10. It extends rather than replaces the OWASP Top 10 for LLM Applications.
2. The primary vector remains ASI01 Agent Goal Hijack: an attacker redirects the agent’s objective through content it reads, whether a document, a review, a message or a tool result.
3. Single-shot probes are insufficient. You have to test the execution loop, because escalation happens across a chain of tool calls rather than in one response.
4. An effective programme combines four levels: broad scanning, regression tests in the CI pipeline, multi-turn adversarial campaigns, and expert manual testing aimed at business logic.
5. Tools cover different layers and have disjoint blind spots: model scanners do not understand an agent’s tool graph or its integrations, and application testing does not substitute for assessing the model itself.
6. The pass criterion is not “was the model persuaded” but “what did the agent actually do, and was there a control outside its reasoning capable of stopping it”.
7. Patronusec runs security testing for environments with an AI component, combining manual testing with automation under our penetration testing service.

---

---

Why is model testing insufficient for assessing agent resilience?

Because the vulnerability does not live in the model but in the permission architecture around it. A model can be talked into generating any text; harm arises only once that text becomes a tool call with an effect in a real system. A scanner querying a model endpoint never sees that boundary.

Three agent properties that change the testing model:

• A loop rather than a single response. The agent plans, calls a tool, reads the result and plans again. Injection can arrive in a tool result at step three and surface as an effect at step seven.
• Persistent memory. An instruction written into long-term memory influences later sessions, so a test that ends with the session misses the persisted effect.
• Blast radius equal to the sum of permissions. An agent’s exposure equals every credential, tool and interface it can reach, and multi-step autonomy compounds damage across a plan.

The OWASP GenAI Security Project describes a configuration where an agent combines three properties at once: access to private data, exposure to untrusted content, and the ability to communicate externally. That combination lets a single injected instruction turn the agent into an exfiltration tool. It is the first thing we look for when mapping an architecture.

You are not testing what the model wrote. You are testing what the agent did, and whether a control outside its reasoning was able to stop it.

How do you scope testing against the OWASP ASI Top 10?

Test scope derives from the ASI taxonomy, because each entry describes a different route from an injected instruction to an effect. Prompt injection functions there not as one category but as the mechanism that triggers most of the others.

OWASP ASI category	What you actually test	Typical evidence of vulnerability
ASI01 Agent Goal Hijack	whether content the agent reads changes its objective	agent pursues a task other than the one assigned
ASI02 Tool Misuse & Exploitation	whether a tool can be forced into attacker-chosen arguments	API call with substituted recipient or scope
ASI03 Identity & Privilege Abuse	whether the agent acts with more privilege than the task needs	action that cannot be attributed to a specific user
ASI04 Agentic Supply Chain	whether a poisoned component, plugin or tool server alters behaviour	instruction executed from a tool description
ASI05 Unexpected Code Execution	whether the agent executes code beyond its intended scope	command executed in the agent environment
ASI06 Memory & Context Poisoning	whether an instruction persists between sessions	effect visible in a fresh session with no re-injection
ASI07 Insecure Inter-Agent Communication	whether a message from another agent is treated as trusted	escalation through agent-to-agent traffic
ASI08 Cascading Failures	whether one agent’s failure propagates	chain reaction from a single injection
ASI09 Human-Agent Trust Exploitation	whether the interface shows the user the full effect of an action	approval given on an incomplete presentation
ASI10 Rogue Agents	whether an agent operates outside oversight	activity with no owner and no record

The ASI taxonomy extends the OWASP list for LLM applications, so an agentic system also inherits model-level risks. In practice that means two test scopes run in parallel: the model layer and the system layer.

How do you build a five-layer testing programme?

An effective programme stacks into layers of increasing cost and increasing depth. Each layer finds a different class of problem, and skipping any one leaves a gap the others will not close.

Layer 1 – architecture mapping (before any testing). An inventory of the agent’s tools, untrusted content sources, memory stores and permission boundaries. Without that map, testing fires blind and the report cannot be translated into fixes. It is the one layer that cannot be automated.

Layer 2 – broad model-layer scanning. Automated probes covering known model vulnerability classes, run on a cycle and on every model change. It catches regressions and low-hanging fruit. The limitation is well known: scanners test models and do not understand your RAG pipeline, tool graph or integrations.

Layer 3 – regression tests in the CI pipeline. A maintained case set acting as a release gate, run on every change to the system prompt, the model, the tool set or the permission configuration. The goal is not discovering new attacks but ensuring fixed issues stay fixed.

Layer 4 – multi-turn campaigns. Scenarios built across many conversational turns and many tool calls, including gradually escalating attacks and sequences that exploit memory. This layer finds the chains single-shot probes cannot see.

Layer 5 – manual testing aimed at business logic. A human with domain knowledge hunts for abuses no tool anticipates: contradictions in business rules, procedural abuse, paths around approval steps. This is where the most expensive and hardest-to-fix problems surface.

Patronusec Insight: The most common failure in testing programmes is starting at layer two without layer one. The team receives a report with hundreds of model-layer findings, none of which says whether the agent can move funds, delete a record or send a document outside the organisation. The reverse order produces results immediately: a permission map usually surfaces three or four paths with a real effect, and testing then confirms or rules out their feasibility. In assessments of environments with an AI component we always begin with the tool graph and the trust boundaries, and only then reach for probes. We select verification methods along the lines set out in our asset on IT security testing.

Which test scenarios are mandatory?

The minimum set covers six scenarios reflecting the routes by which an injected instruction actually reaches an agent. Each needs a different delivery channel and a different way of observing the effect.

1. Injection through retrieved content. An instruction hidden in a document, a web page, a product description or a review. You observe whether the agent changes objective after reading it.
2. Injection through a tool result. An instruction inside an API response or in metadata returned by an external service. This channel is routinely skipped, because teams treat their own integrations’ output as trusted.
3. Injection through a tool description. An instruction placed in the description of a function exposed to the agent by an external tool server. Agents read those descriptions as part of their context.
4. Memory poisoning. An instruction persisted into long-term memory, then tested in a fresh, clean session. The pass criterion: does the effect occur without re-injection.
5. Exfiltration through a permitted channel. An attempt to move data out through an integration the agent legitimately holds, such as sending a message or writing to a shared resource.
6. Abuse of the user-facing presentation. Checking whether the approval interface shows the full effect of an action or only a summary. If the user approves only an amount or a title, that approval is not a real control.

For each scenario, record three things: the exact injection payload, the complete tool-call trace, and the actual effect in the target system. A model repeating an instruction is not, by itself, a vulnerability. The executed action is.

How do you interpret results, and what separates a fix from a workaround?

Findings should be classified by their effect in the system rather than by the model’s behaviour. A durable fix operates outside the agent’s reasoning; a workaround adds another sentence to the system prompt and survives only until the next attack variant.

The hierarchy of fixes, from most to least durable:

• Permission reduction. Removing a tool or narrowing its scope eliminates an entire class of scenarios regardless of attacker ingenuity.
• Context separation. Splitting the path that processes untrusted content from the path holding access to sensitive data and the ability to act.
• Controls outside the agent. Validation of call parameters, limits, human approval for irreversible actions, and checks that an action matches the original instruction.
• Input and output filtering. Useful as an additional layer, unreliable as the only one.
• Hardening the system prompt. Cheapest and least durable. Treat it as a stopgap, never as closure of a finding.

Every fixed vulnerability belongs in the layer-three regression set. Without that step the same issue returns after the next model change and the organisation pays to rediscover it.

A fix that consists of adding a sentence to the system prompt is a hypothesis, not a control. A control is a constraint that holds regardless of what the agent decides makes sense.

What is most often misunderstood about prompt injection resilience?

MYTH: A good model is enough, because newer models resist injected instructions.
FACT: Model robustness reduces the success rate of simple attempts but does not remove the class of problem. The effect depends on the agent’s permissions, not on how hard it is to persuade.

MYTH: Guardrails solve it.
FACT: Input and output filters raise the cost of attack and are worth having, but they act on content rather than on permissions. Against an attack run across many turns and many tool calls they remain a supplementary layer.

MYTH: If the agent has no access to sensitive data, the risk is low.
FACT: Risk is set by the sum of permissions, including the ability to act. An agent with no data access but with the right to invoke operations can cause harm without reading anything sensitive.

MYTH: A test before deployment closes the subject.
FACT: Changing the model, adding a tool or editing the system prompt changes the attack surface. Without a regression set run on every such change, a test result expires with the first release after it.

MYTH: Injection only counts when it comes from the user.
FACT: The most dangerous variant is indirect: a third party leaves the instruction in content the agent will read while carrying out a task assigned by an honest user.

FAQ – Testing AI agent resilience to prompt injection

---

What to read next

A path from understanding the risk to implementing controls:

1. IT security testing – how to choose the right method – selecting a verification method for an environment with an AI component.
2. Agentic payment security – risks for 5 market roles – the business consequences of manipulating an agent’s intent in transactions.
3. AI usage policy for companies – 10 critical elements – the governance layer that gives agent controls an owner.
4. AI-enhanced attacks – how AI changes cyber threats – the wider context of offensive AI use.

---

AI agent resilience testing – free consultation

Patronusec runs security testing for environments with an AI component, combining architecture mapping with manual testing and automation.

Book a free consultation | Penetration testing | Vulnerability scans