Prompt injection defense is an application-security workflow, not a single prompt rule. This guide shows how to threat-model direct and indirect attacks, limit what retrieved content and models can do, test defenses with repeatable cases, and establish monthly and quarterly checkpoints for RAG applications, chatbots, and AI agents.
Overview
A prompt injection attack occurs when untrusted text influences an AI system to ignore, alter, or work around the instructions and boundaries defined by the application. The text may come directly from a user, or it may be hidden in a document, web page, email, ticket, image, or database record that the application retrieves later. In an agent workflow, the consequences can extend beyond an incorrect answer: the model may select an unsafe tool, expose data, modify a record, or trigger an external action.
The central security assumption is simple: anything that enters the model from outside the trusted application policy is data, not authority. This includes retrieved passages, tool results, conversation history supplied by a client, uploaded files, and text generated by another model. Clear instruction hierarchy can help the model distinguish roles, but it should not be treated as a complete security boundary. Enforcement belongs in application code, access controls, tool design, and operational monitoring.
A useful defense workflow has five parts:
- Map trusted instructions, untrusted inputs, data stores, tools, and possible side effects.
- Separate content for analysis from instructions that control behavior.
- Give the model the minimum tool access and data access required for the task.
- Validate inputs and outputs outside the model wherever practical.
- Run repeatable adversarial tests and review logs after application changes.
For a broader application checklist, see Prompt Injection Prevention: A Security Checklist for LLM Applications. This article focuses on the recurring workflow for monitoring and improving those controls.
What to track
1. Trust boundaries and attack paths
Document every place where instructions or data cross into the AI workflow. At minimum, record the system policy, developer instructions, user messages, conversation memory, retrieved documents, tool responses, uploaded files, and output destinations. For each source, label whether it is trusted, partially trusted, or untrusted. Also note who can write to it and whether its contents are later retrieved for another user.
This inventory often reveals indirect prompt injection risks. A knowledge-base article may look harmless during ingestion but contain text that tells the model to reveal hidden instructions. A support ticket may be retrieved into an agent that can send email. A web page may contain instructions that conflict with the task the user actually requested. Treat the entire retrieval and tool chain as an attack surface, not just the chat box.
2. Model authority versus application authority
Track which decisions the model may suggest and which decisions the application enforces. The model can propose a search query, but the server should enforce the user’s permissions. The model can draft a database update, but a separate validation layer should check the fields, scope, and target before execution. The model can request a tool, but the tool gateway should verify authorization, arguments, rate limits, and the need for human approval.
Useful fields in an internal control register include:
- Tool name and purpose
- Permitted user roles
- Read, write, delete, or external-communication capability
- Allowed argument types and value ranges
- Whether execution requires confirmation
- Logging and rollback provisions
Keep tools narrow. A function that performs one constrained operation is easier to authorize and test than a general-purpose tool that accepts arbitrary code, URLs, queries, or filesystem paths. The Function Calling and Tool Use guide provides a useful implementation companion.
3. Input, retrieval, and output signals
Track more than blocked phrases. A robust review looks for behavioral signals such as attempts to override task constraints, requests for hidden prompts or secrets, suspicious instructions inside retrieved content, unusual tool sequences, repeated authorization failures, and outputs that contain credentials, personal data, or unapproved commands.
Input and output filters should support the design, not replace it. Keyword rules can miss paraphrases and can also block legitimate content. Prefer structured checks where possible: validate JSON against a schema, allow only known tool names, reject unexpected fields, constrain URLs and query scopes, and apply authorization independently of the model. See Structured Outputs for LLMs for a related validation pattern.
Attack-scenario comparison
| Scenario | Typical entry point | Primary risk | Priority controls |
|---|---|---|---|
| Direct injection | User message | Policy bypass, data disclosure, unsafe response | Task boundaries, output checks, refusal tests, rate limits |
| Indirect injection | Retrieved document or web content | Untrusted text changes the model’s behavior | Source labeling, retrieval isolation, content handling rules, red-team cases |
| Tool manipulation | Model-generated function call | Unauthorized read, write, deletion, or external action | Server-side authorization, strict schemas, least privilege, approval gates |
| Memory poisoning | Conversation or persistent memory | Malicious instructions persist across sessions | Memory filtering, provenance, expiry, user-scoped storage, review controls |
| Data exfiltration | Prompt, retrieval, or tool output | Secrets or private records appear in a response | Access control, secret redaction, retrieval scoping, egress monitoring |
Cadence and checkpoints
Security review works best as a recurring operating process. A monthly review can be lightweight; a quarterly review should be deeper and involve application, platform, and security owners.
Monthly checkpoint
- Review blocked and allowed injection attempts, including false positives.
- Sample retrieved documents and tool results for embedded instructions.
- Confirm that tool permissions and approval requirements still match the product’s behavior.
- Run a small regression set covering direct injection, indirect injection, data disclosure, and unsafe tool calls.
- Check whether new data sources, integrations, memory features, or prompt changes were introduced.
- Verify that security logs contain request identifiers, model version identifiers, tool decisions, and outcomes without storing unnecessary sensitive content.
Quarterly checkpoint
- Rebuild the threat model for material workflow changes.
- Review every tool’s least-privilege design and test denied actions.
- Expand the red-team set with newly observed attack patterns and realistic business data.
- Test fallback models, model migrations, retrieval changes, and degraded-service paths.
- Review incident response: who can disable a tool, revoke access, preserve logs, and communicate an impact?
- Reassess provider controls, framework guidance, and relevant security practices when platform behavior changes.
Keep these checkpoints connected to release management. A prompt-only change may deserve regression testing; adding browsing, email, payments, code execution, or write access deserves a full security review. The AI Agent Workflow Design guide can help map approval and isolation patterns.
How to interpret changes
An increase in detected attacks does not automatically mean the application became less secure. It may reflect higher traffic, better instrumentation, a new data source, or a wider test set. Interpret trends alongside exposure: number of sessions, retrieval volume, tool calls, blocked events, confirmed incidents, and false-positive reviews.
Separate four outcomes:
- Attempt blocked safely: the system refused or contained the request and did not expose data or perform an unsafe action.
- Attempt detected but poorly handled: the system recognized suspicious content but produced confusing output, leaked partial context, or created excessive operational noise.
- Control bypass: the model or workflow crossed a policy boundary, even if no harm followed.
- Business-impacting incident: unauthorized data access, state change, external communication, or service disruption occurred.
Do not measure success only by refusal rate. A model that refuses every request is not a usable defense. Track task completion for legitimate cases, safe handling of adversarial cases, tool-call correctness, and the ability to explain or audit decisions. Prompt testing should include both normal and hostile examples; the Prompt Testing Playbook and Prompt Testing and Evaluation framework offer reusable structures for this work.
Reusable red-team test case
Store each test as a structured record so it can run again after a model, prompt, retrieval, or tool change:
Test ID:
Workflow and version:
Attacker entry point: direct | document | web | memory | tool result
User role and permissions:
Untrusted content:
Expected safe behavior:
Forbidden data or action:
Tools that must remain unused:
Observed response:
Observed tool calls:
Pass/fail:
Evidence and follow-up owner:Use synthetic secrets and reversible test actions. Never make a production red-team case depend on sending real messages, changing real records, or exposing live credentials.
When to revisit
Revisit this guide at least monthly for operational signals and quarterly for a broader security review. Return sooner whenever the application adds a tool, data source, browsing capability, persistent memory, file upload, external integration, or write operation. A model migration, prompt rewrite, retrieval-index rebuild, authentication change, or new user population is also a trigger.
Before each significant release, run the red-team cases that match the changed components. If retrieval changed, emphasize indirect injection and source permissions. If tools changed, test argument validation, authorization, confirmation, and rollback. If prompts or models changed, compare both safety and legitimate-task performance. If logs or filters changed, verify that investigators can still reconstruct the event without collecting more sensitive data than necessary.
To make the process actionable, assign one owner to the security checklist, one owner to tool permissions, and one owner to test maintenance. Record the last review date, the next scheduled checkpoint, unresolved risks, and the conditions that would require an immediate review. A maintained test set and a clear change log turn prompt injection defense from a one-time exercise into a durable part of AI development workflows.
