1.4Key Concepts and Terminology

Before diving into specific vulnerabilities, establish a common vocabulary:

Prompt: The text input sent to an LLM. Includes the system prompt (developer-defined instructions), conversation history, and user input.

System prompt: The developer-defined instructions that establish the LLM's behavior, persona, restrictions, and capabilities. This is the first line of defense.

Prompt injection: An attack where the attacker's input manipulates the LLM into ignoring its system prompt and following the attacker's instructions instead.

Jailbreak: A specific form of prompt injection aimed at bypassing the model's safety guidelines to produce content the model is designed to refuse.

RAG (Retrieval-Augmented Generation): An architecture where the LLM's response is augmented with information retrieved from external documents or databases.

Tool use / Function calling: The ability of an LLM to invoke external functions (APIs, database queries, file operations) as part of generating a response.

Guardrails: Safety mechanisms that constrain the LLM's behavior, including input filters, output validators, and content classifiers.

Red teaming: Adversarial testing where skilled human testers attempt to find vulnerabilities through creative, unscripted attacks.

Defense in depth: The principle of using multiple overlapping security layers so that if one fails, others still protect the system.