8 / 70 · 07 Security Testing for AI Apps · Insecure Output Handling and Model Denial of Service← prev ⊞ all next →☰ Read as one page

2.2The Core Problem

Traditional input validation protects against user-supplied malicious input. But when an LLM generates the output, developers often skip validation because they think of the AI as a trusted internal component. This is a dangerous assumption -- the model's output is influenced by user input (via the prompt) and can contain malicious content.

Attack Flow

User Input (potentially malicious)
    |
    v
[LLM generates response influenced by user input]
    |
    v
[Application uses LLM output in downstream operation]
    |
    +---> SQL query (SQL injection via AI)
    +---> Shell command (OS command injection via AI)
    +---> HTML template (XSS via AI)
    +---> API call parameters (SSRF via AI)
    +---> File path (path traversal via AI)