Testing AI agents – the most common mistakes, OWASP Top 10 for LLM and Agentic, and how to test models in-house

What does testing AI agents involve? Testing AI agents means verifying the security of systems in which a language model does not merely generate text but plans actions, calls tools (APIs, databases, a browser, a terminal), keeps memory and makes decisions with limited human oversight.

The reference points are two OWASP lists: the Top 10 for LLM Applications 2025 (application-layer risks, from prompt injection to unbounded consumption) and the Top 10 for Agentic Applications 2026 (ASI01-ASI10 – autonomy-specific risks: goal hijack, tool misuse, memory poisoning, rogue agents). This article is written for CTOs: what most often goes wrong when building agents, how to test in-house, and where the work of an external red team begins.

Canonical definition: AI agent security testing is the assessment of a system built on a language model equipped with tools, memory and decision-making autonomy, covering resistance to instruction manipulation, abuse of privileges and tools, context poisoning, and the consequences of actions executed without human oversight.

Testing AI agents – key takeaways:

1. An agent differs from a chatbot in that it acts rather than merely answers – so the outcome of an attack is not a bad reply but a sent email, a dropped database or a transfer; the threat model must be built around tools and privileges, not content.
2. The OWASP Top 10 for LLM Applications 2025 (v2.0) describes model- and application-layer risks – with prompt injection (LLM01) still in first place; the OWASP Top 10 for Agentic Applications 2026 (ASI01-ASI10, published 9 Dec 2025) adds the risks of autonomy.
3. The most common architectural mistake is excessive agency: an agent with “just in case” privileges, a shared service identity and tools broader than the task requires – against the Least Agency principle.
4. In-house testing is feasible: LLM vulnerability scanners (garak), red-teaming frameworks (PyRIT) and prompt regression tests in CI/CD (promptfoo) cover known attack classes – but not business logic or an agent’s action chains.
5. Prompt injection cannot be “patched” like a CVE – the model processes instructions and data in one channel; the defence is architectural: content separation, output validation, approval gates, sandboxing and blast-radius limits.
6. Patronusec includes LLM application and agent security testing in its penetration testing – scenarios built to both OWASP lists, from indirect prompt injection to tool misuse, with a report mapped to specific LLM01-LLM10 and ASI01-ASI10 identifiers.

---

---

How does an AI agent differ from an LLM application, and why does that change the threat model?

An LLM application generates answers; an AI agent plans and executes: it breaks a goal into steps, calls tools, writes conclusions to memory and iterates without asking a human at every step. That difference changes the threat model in three dimensions: the result of an attack is an action rather than text, the attack surface becomes everything the agent reads (documents, pages, emails, its own memory), and errors propagate through the step chain and between agents.

Dimension	LLM application (chat, RAG)	AI agent
Model output	text for a human	tool calls: API, SQL, email, code, browser
Result of a successful attack	a harmful or wrong answer	an executed action: exfiltration, data change, transaction
Attack surface	user prompt, RAG documents	+ tool results, pages the agent browses, persistent memory, other agents’ messages
Human oversight	at every answer	limited – the agent runs a plan-act-observe loop
Error propagation	ends with the answer	cascades: a poisoned step 2 shapes steps 3-10
Identity	the user asks on their own behalf	the agent acts with privileges – often broader than the user’s

In an agentic system, every piece of content the agent reads – a document, a web page, a tool result, an entry in its own memory – is a potential instruction channel for an attacker, so the agent’s attack surface is its entire input stream, not just the user prompt.

The practical consequence for a CTO: “is the model secure” is the wrong question. The right questions are: what can the agent do (tools, privileges), influenced by what (all content sources), and what happens when it does it wrong (blast radius). Testing an agent means verifying those three answers, not benchmarking model quality.

What mistakes do teams building AI agents make most often?

The most common mistakes are not about model choice but about the architecture around it: excessive privileges, treating external content as trusted, executing generated code without isolation, a shared identity for the agent, and no limits on the action loop.

❌ The agent on a broad service account – one token for everything (mailbox, CRM, database), shared across tasks; after a successful injection the attacker inherits it all (ASI03). Correct: identity per agent and per task, short-lived, narrowly scoped credentials.

❌ External content flowing straight into the decision context – the agent reads an email, page or PDF and follows instructions hidden inside (indirect prompt injection → ASI01). The classic OWASP example: an email-summarising plugin that also has a send function – a malicious email makes the agent send spam. Correct: retrieved content marked as data, not instructions; separation patterns; no “write” tools in the same run in which the agent reads untrusted sources.

❌ Executing generated code and commands without a sandbox – the agent “helpfully” runs a script on a host with internal network access (ASI05, LLM05). Correct: isolated execution, egress blocked by default, time and resource limits.

❌ Persistent memory without write validation – one injected “note” poisons all future sessions of this agent and any agents sharing the same vector store (ASI06, LLM08). Correct: validated writes with provenance, per-task rather than global memory, the ability to purge.

❌ No loop or budget limits – an agent in a retry loop generates thousands of API or model calls (LLM10, ASI08 – cascade). Correct: step, cost and time limits, a kill switch.

❌ Human approval based on the agent’s summary – the user clicks “OK” on an action description written by the agent, not on the raw action (ASI09). Correct: the gate shows the exact call (recipient, amount, SQL statement), not the narrative.

❌ Testing only the happy path before launch – the demo works, so production ships; nobody tried injection via data, goal conflicts, or a tool failing mid-chain.

Most AI agent incidents result not from model weakness but from architectural decisions made around it: excessive privileges, trust in external content, and no isolation of executed actions.

What does the OWASP Top 10 for LLM Applications 2025 cover, and which risks also apply to agents?

The OWASP Top 10 for LLM Applications 2025 (v2.0, published 18 Nov 2024 by the OWASP GenAI Security Project) organises ten model- and application-layer risks – all of which apply to agents too, because an agent is an LLM application with an extended surface. The additions versus the 2023 edition reflect production practice: system prompt leakage (LLM07), vector and embedding weaknesses in RAG (LLM08) and unbounded consumption (LLM10).

ID	Risk	Essence	Relevance to agents
LLM01	Prompt Injection	input (direct or hidden in content) alters model behaviour	the main vector for hijacking an agent’s goal; multimodal variant – instructions in images
LLM02	Sensitive Information Disclosure	secrets, PII, business data leak via outputs or traces	an agent with system access has plenty to leak; exfiltration via tools (e.g. an HTTP request with data in the URL)
LLM03	Supply Chain	compromised models, adapters, dependencies	+ MCP servers, tool definitions, dynamically fetched plugins
LLM04	Data and Model Poisoning	poisoned training or fine-tuning data	a poisoned RAG corpus acts as a persistent injection
LLM05	Improper Output Handling	model output executed without validation (XSS, SSRF, RCE)	an agent’s output is tool calls – parameter validation is critical
LLM06	Excessive Agency	too much functionality, permission or autonomy	expanded in 2025 precisely because of agentic architectures; the foundation of ASI02-ASI03
LLM07	System Prompt Leakage	disclosure of the system prompt and secrets within it	reveals the agent’s tools, constraints and how to bypass them
LLM08	Vector and Embedding Weaknesses	manipulation of the vector store and retrieval	agent memory usually sits on the same infrastructure
LLM09	Misinformation	confabulations taken as facts	the agent acts on confabulations (e.g. installs a non-existent, squatted package)
LLM10	Unbounded Consumption	resource- and budget-exhausting queries; model extraction	agent loops multiply cost; a DoS vector

Prompt injection has held first place in the OWASP Top 10 for LLM applications since the first edition, because the model processes instructions and data in a single language channel – no single control closes this attack class, and the defence must be layered.

One more observation from the OWASP document matters for delivery teams: the risks chain. The typical production scenario is LLM01 → LLM05/LLM06: an injected instruction (via a RAG document) makes the model generate output that the application executes with the agent’s privileges. Tests must therefore exercise chains, not isolated categories.

What does the OWASP Top 10 for Agentic Applications 2026 (ASI01-ASI10) add?

The OWASP Top 10 for Agentic Applications 2026 – published 9 Dec 2025 by the OWASP GenAI Security Project, peer-reviewed by more than 100 experts and announced at Black Hat Europe 2025 – catalogues ten risks specific to autonomous systems: those that plan, use tools, hold memory and coordinate with other agents. The architectural principle binding the whole list is Least Agency: grant an agent only the minimum autonomy required for a safe, bounded task.

ID	Risk	Essence	First line of defence	Related LLM Top 10
ASI01	Agent Goal Hijack	attackers alter the agent’s objectives through malicious content	retrieved content = untrusted; bounded, explicit goals	LLM01
ASI02	Tool Misuse and Exploitation	legitimate tools used in unsafe ways	per-task tool scoping; parameter validation	LLM06
ASI03	Identity and Privilege Abuse	the agent inherits or escalates privileges	per-agent identity; short-lived scoped credentials	LLM06
ASI04	Agentic Supply Chain	compromised tools, plugins, MCP servers	signed components; AIBOM and provenance; version pinning	LLM03
ASI05	Unexpected Code Execution	the agent generates or runs code/commands unsafely	sandboxed execution; deny-by-default egress	LLM01, LLM05
ASI06	Memory and Context Poisoning	poisoned agent memory and RAG stores shape future decisions	validated memory writes; ephemeral per-task context	LLM04, LLM08
ASI07	Insecure Inter-Agent Communication	spoofing and tampering in multi-agent systems	mutual authentication; signed messages	–
ASI08	Cascading Failures	a small error propagates through planning and execution	blast-radius isolation; circuit breakers	–
ASI09	Human-Agent Trust Exploitation	users over-trust the agent’s recommendations	forced confirmations showing the raw action	LLM09
ASI10	Rogue Agents	a compromised agent acts harmfully while looking legitimate	behavioural monitoring; kill switches	–

The Least Agency principle – granting an agent only the minimum autonomy required for a safe, bounded task – is the architectural foundation of the entire OWASP Top 10 for Agentic Applications 2026.

Note the three entries with no LLM-list counterpart, because they define the “new” testing work: ASI07 (inter-agent communication – who authenticates a message “from the approval agent”?), ASI08 (cascades – one poisoned state triggers destructive actions across multiple agents) and ASI10 (a rogue agent persisting across sessions and impersonating trusted systems). Every entry on the list is tied to a documented real-world incident – from hidden prompts turning copilots into silent exfiltration channels to an autonomous coding agent deleting a production database during a code freeze.

How do you test models and agents in-house – tools and process?

In-house testing rests on three layers: automated LLM vulnerability scanners run periodically (garak), frameworks for semi-automated scenario red teaming (PyRIT), and prompt and policy regression tests in the CI/CD pipeline (promptfoo), complemented by the team’s manual adversarial sessions. The principle is the same as with application scanning: automation covers the known classes, humans cover logic and chains.

Layer	Tools (open source)	What it covers	What it does not	When to run
LLM vulnerability scan	garak (NVIDIA) – dozens of probes: jailbreaks, leaks, toxicity, package hallucinations	known attack patterns against the model and system prompt	business context, the agent’s tools	periodically and after any model or system prompt change
Scenario red teaming	PyRIT (Microsoft) – multi-step attack orchestration, payload conversions; giskard – scanning + tests	adaptive attacks, sequences, encoding variants	full impact assessment on target systems	before launch and at significant changes
CI/CD regression	promptfoo – output assertions, built-in red-team plugins mapped to the OWASP LLM and Agentic lists	ensuring a fix does not regress with the next release	new, unknown attack classes	every build / pull request
Manual testing	team adversarial sessions + OWASP AI Testing Guide scenarios	business logic, LLM01→LLM06 chains, tool misuse	scale and consistency	before launch, quarterly

A process that works in practice (and can be stood up by the team in 2-4 weeks):

1. A threat model per agent – the list of tools with privileges, content sources and worst-case impact; ASI01-ASI10 mapped as a design checklist.
2. A baseline test set – garak against the model/application endpoint; results as the reference point.
3. Agentic scenarios – for every tool: “what happens if input content tells the agent to use it differently?”; indirect injection payloads in test documents, emails and pages; exfiltration attempts via tool parameters (URL, recipient, query body).
4. CI/CD gates – promptfoo with blocking thresholds: no prompt, model or tool definition change ships without passing the suite; the same logic as SAST/SCA gates.
5. A findings register – every vulnerability with an OWASP identifier, owner and deadline, in the same process as infrastructure vulnerability management.

Automated LLM vulnerability scanners cover known attack patterns, but they will not answer what your agent will do with your tools under hostile content – that answer comes from scenarios built out of the specific agent’s threat model.

Patronusec Insight: In our project experience, teams that start from tools (“let’s run a scanner”) get stuck on hundreds of low-value findings, while teams that start from a threat model find critical issues in days. The most effective first exercise needs no tool at all: take your agent’s tool list and, for each tool, finish the sentence “an attacker who controls any document the agent reads can use this tool to…”. If any answer includes “send”, “delete” or “transfer” without an approval gate – you have your priority before the first scan runs. In our penetration tests of LLM applications, we start with exactly this exercise together with the client’s team.

Which architectural practices most effectively limit agent risk?

Effective defence is layered and architectural, because prompt injection has no patch: separating instructions from data, least agency in tools and identity, output validation before execution, human gates on irreversible actions, sandboxing with egress control for code, validated and ephemeral memory, and behavioural monitoring with a kill switch.

Eight practices in implementation order:

1. Least Agency in tools – tools defined per task, not per agent; parameters validated by schema (allowed domain lists, amount limits, table whitelists); read and write operations separated.
2. Identity per agent and per task – no shared service accounts; short-lived, narrowly scoped, rotated credentials; agent actions attributable in logs (ASI03, ASI07).
3. Separation of untrusted content – retrieved content (RAG, web, email) wrapped and marked as data; a “read-only in runs that touch untrusted sources” policy – an agent that reads the internet and can send data in the same chain is by definition an exfiltration channel.
4. Output validation before execution – model output treated like any untrusted input (LLM05): structural parsing, rejection of calls outside the schema, sanitisation before rendering.
5. Human gates showing the raw action – for irreversible or financial actions: the exact call (recipient, amount, command), not the agent’s summary (ASI09); thresholds and categories requiring approval written into policy, not the prompt.
6. Sandbox and egress control for code – isolated execution, outbound network denied by default with an exception list, CPU/time/disk limits (ASI05).
7. Memory with provenance and lifetime – validated writes tagged with source, per-task where possible; periodic purging; memory-read tests in scenarios (ASI06).
8. Limits, monitoring, kill switch – step/cost budgets per run, alerts on behavioural anomalies (a sudden shift in tool-call patterns), a documented kill switch and agent isolation procedure (ASI08, ASI10, LLM10).

Defending AI agents is not about perfecting the system prompt but about an architecture that limits consequences: an agent whose worst possible mistake is reversible and visible remains safe even when manipulation of the model succeeds.

Two structuring notes for a CTO. First, agentic components are also a classic software supply chain: MCP servers, tool definitions and models deserve the same treatment as libraries – inventory (AIBOM/SBOM), version pinning, provenance verification – i.e. the practices known from SCA and SBOM and the obligations the CRA places on manufacturers. Second, if the organisation falls under the AI Act, the agent’s threat-model and test documentation will feed the high-risk system obligations – design it in an evidence-ready format from the start.

When is in-house testing enough, and when do you need external AI red teaming?

In-house testing suffices for continuous hygiene: regression of known attacks, CI/CD gates and verification of basic tool-misuse scenarios. An external test is needed in four situations: before exposing the agent to customers or production data, when the agent gains tools with irreversible consequences, when a customer or regulation demands it, and when the internal team is testing its own design – and has stopped seeing its assumptions.

The boundary runs along three questions:

• Can we think like an attacker of our own system? The team that built the agent tests within its own assumptions; an external tester breaks them – the same logic for which a pentest is not a scan.
• Are the consequences of a mistake reversible? An agent drafting replies tolerates iteration in production; an agent with access to transfers, the production database or the executive mailbox does not.
• Does anyone require it of us? Customer due diligence questionnaires increasingly contain an AI section; for regulated entities, testing of applications – including LLM-based ones – falls under the DORA Article 24-25 testing programme and ISO 27001 risk analysis.

An external AI agent security test is the counterpart of an application penetration test: it verifies attack chains and logic that automated scanners do not model, and delivers a report you can hand to a customer or an auditor.

Patronusec delivers LLM application and agent security testing as a module of its penetration testing: scenarios built from the client’s threat model, attacks to the OWASP Top 10 LLM 2025 and Agentic 2026 (indirect injection via RAG and email, tool misuse and chaining, exfiltration via parameters, memory poisoning, approval-gate bypasses), and a report mapping every finding to an LLM/ASI identifier with an architectural recommendation – in a format ready for audits and customer questionnaires.

CTO checklist – 12 control questions before deploying an agent

1. Is there a written threat model for the agent: tools, privileges, content sources, worst case per tool?
2. Does every agent have a separate identity with short-lived, minimally scoped credentials?
3. Can the agent read untrusted content and perform write/send actions in the same run? If so – why?
4. Are every tool’s parameters validated by schema (domains, amounts, ranges) before execution?
5. Do irreversible actions pass through a gate showing the raw action, with thresholds in policy rather than the prompt?
6. Does generated code run in a sandbox with egress denied by default?
7. Are writes to persistent memory validated and tagged with provenance, and can the memory be purged?
8. Is inter-agent communication mutually authenticated?
9. Are there step, cost and time limits per run, and a documented kill switch?
10. Do prompt and policy regression tests act as a gate in CI/CD?
11. Are agentic components (models, MCP servers, tools) inventoried, pinned and verified like SBOM dependencies?
12. Before customer exposure, is an independent security test planned, and will its report feed compliance documentation?

---

FAQ – common questions about testing AI agents

---

Recommended reading

1. IT security testing methods – where AI agent testing sits in the overall testing programme alongside pentests and scans.
2. Application vulnerability scanning – DAST, SAST, SCA – CI/CD gates and SBOM, whose logic carries straight over to agentic components.
3. CRA for software houses – manufacturer obligations that will also cover products with AI components.
4. Malicious NuGet package StripeAPI.NET – the supply chain in practice; the same risk class as ASI04 and hallucinated packages (LLM09).

---

AI agent security testing – consultation

Patronusec delivers LLM application and AI agent security testing as a module of penetration testing: a threat-model workshop, scenarios to the OWASP Top 10 LLM 2025 and Agentic 2026, and a report mapped to LLM/ASI identifiers, ready for audits and customer questionnaires. If you are deploying an agent with access to real tools and data, book a free scoping call.

Penetration testing | Vulnerability scanning | vCISO

---