The first time most security organizations red-team an AI system, they hand the engagement to their existing pentest provider, scope it like a web application, and ask for the same deliverable they always ask for. The report comes back, the AI system passes, and the team moves on. Six months later, an internal user discovers a prompt injection that exfiltrates data the system was never supposed to expose, and the question becomes: what did the pentest actually test? The honest answer is that it tested the API, the auth layer, and the infrastructure, the parts a traditional pentest is built to evaluate. The AI part went largely untouched.
What Traditional Pentesting Is Actually Testing
A traditional pentest of an AI-powered application follows the methodology built for web applications: enumerate the attack surface, look for OWASP Top 10 issues, test authentication and authorization, probe for injection vulnerabilities in the conventional sense, and check the supporting infrastructure. This is real work, and the findings are real. They are also almost entirely orthogonal to the risks introduced by the AI component.
The reason this matters: AI systems concentrate risk in places traditional methodologies don't probe. The model itself, the prompt construction logic, the retrieved context, the tool integrations, the agent's decision-making graph, these are the surfaces where AI-specific vulnerabilities live, and a methodology built for stateless HTTP request-response cycles will miss them.
What AI Red-Teaming Actually Tests
Prompt injection across all entry points. Direct user input, but also every channel through which untrusted content reaches the model: retrieved documents, indexed content, third-party API responses, content uploaded by other users. A real AI red-team maps every input source, classifies trust, and tests injection through each one. Most of the high-impact findings come from indirect injection paths the development team didn't realize existed.
Context boundary violations. AI systems are typically scoped to a particular role and dataset. Red-teaming probes whether the model can be coerced into operating outside that scope, accessing data from other users, executing actions intended for administrators, or bridging between contexts that should be isolated. The vulnerability isn't always in the model. It's often in the prompt construction logic that decides what context to provide.
Tool misuse. Agents with access to tools have an expanded attack surface defined by what those tools can do. Red-teaming evaluates whether each tool can be invoked outside its intended use case, whether arguments can be manipulated through the model's reasoning, and whether sequences of tool calls can produce outcomes that no single call would permit.
Data exfiltration paths. Standard exfiltration testing looks for direct data extraction. AI systems open subtler paths: a model trained or fine-tuned on sensitive data may regurgitate it under the right prompts; a retrieval-augmented system may surface sensitive documents in response to crafted queries; an agent with email access may be tricked into sending data outbound through what looks like normal use of its tools. Each path needs to be tested specifically.
Output manipulation. For systems where the model's output drives downstream actions, content moderation decisions, code generation that's executed, financial calculations, red-teaming evaluates whether the output can be shaped to produce a specific malicious result. The vulnerability isn't that the model can be jailbroken. It's that the model's output is trusted by a downstream system that doesn't validate it.
Model and supply chain. Where did the model come from? What's the patch and version control story? Is there evidence that fine-tuning data has been verified for poisoning attacks? For systems using third-party models, what's the contract say about model updates, and how are updates validated before they reach production? Red-teaming includes the supply chain in scope, even when the supply chain is opaque.
Why Methodology Matters More Than Tooling
There is a growing market of automated AI red-teaming tools, prompt fuzzers, jailbreak benchmarks, automated injection scanners. They have their place. They are not a substitute for an engagement led by people who understand the system architecture. The high-impact findings in AI red-teaming come from understanding the specific data flows, trust boundaries, and tool integrations of the system under test, and designing payloads tailored to those specifics. Generic jailbreak benchmarks tell you whether the model is hardened against known patterns. They don't tell you whether your system is secure.
The skill set required is also distinct. AI red-teamers need security background, model behavior intuition, and the ability to think about systems where the attack vector is natural language. People who can do all three are rare, and the engagement quality varies dramatically based on who is staffed.
"A traditional pentest of an AI system is not a wrong test. It's an incomplete one. The findings are real, and so are the gaps the methodology can't see."
What Regulated Environments Should Require
Scope documentation that names AI-specific surfaces. If the engagement scope is "test the application," AI-specific testing will be optional. The scope needs to explicitly include prompt injection (direct and indirect), tool misuse, context boundary testing, data exfiltration paths through the model, and output manipulation. Without explicit scope, the engagement reverts to web-app pentesting by default.
A threat model that reflects AI risk. Generic STRIDE or PASTA models don't capture AI-specific threats well. Use a threat modeling approach that incorporates the OWASP LLM Top 10 or NIST's AI risk taxonomy as inputs, and ensure the threat model is reviewed and updated as the system changes.
Pre-deployment and recurring testing. AI systems evolve. Models update, prompts change, tool integrations expand, retrieved data sources are added. Every meaningful change to any of these should trigger reassessment, because the attack surface shifts with each. Annual testing is insufficient for systems on a rapid development cadence.
A staffing standard that requires AI-specific expertise. "We use a leading pentest firm" doesn't establish that the people on the engagement know how to test AI systems. The contract should specify that personnel have demonstrated AI security expertise, and the report should make clear which findings came from AI-specific testing versus traditional methodology.
The Bottom Line
Red-teaming AI systems is an emerging discipline, the methodologies are not fully standardized, and the talent pool is narrow. None of that is an argument for deferring the work. It is an argument for being deliberate about scope, methodology, and staffing, and for treating AI red-teaming as a distinct capability rather than a checkbox added to an existing engagement. The organizations that get this right will detect their AI vulnerabilities under controlled conditions. The ones that don't will detect them in production, with adversaries setting the agenda.