67 / 70 · 07 Security Testing for AI Apps · Building a Comprehensive AI Security Testing Program← prev⊞ allnext →☰ Read as one page
11.4Red Team Exercises for AI
What Is AI Red Teaming?
AI red teaming is human adversarial testing where skilled testers attempt to break the AI system using creative, unscripted attacks. Unlike automated tests (which check known attack patterns), red teams discover novel vulnerabilities.
Red Team Scope
| Focus Area | Techniques | Duration |
|---|---|---|
| Prompt injection | Creative injection, chained attacks, multi-language | 2-3 days |
| Jailbreaking | Novel persona attacks, context manipulation | 2-3 days |
| Data extraction | PII probing, system prompt extraction, training data recovery | 1-2 days |
| Business logic abuse | Unauthorized actions via AI, social engineering the AI | 1-2 days |
| Traditional web security | Standard pentest with AI endpoint focus | 3-5 days |
Red Team Process
- Scope and rules of engagement: Define what is in-scope, what is off-limits, and the reporting process
- Discovery: Red team explores the AI system, maps its capabilities, and identifies potential attack vectors
- Exploitation: Attempt to exploit identified vulnerabilities
- Reporting: Document findings with severity, reproduction steps, and recommendations
- Remediation: Development team fixes findings
- Verification: Red team verifies fixes and attempts to bypass them