Prompt Injection Defense Checklist for RAG and Tool-Using Apps

Overview

This article gives you a working prompt injection defense checklist for RAG and tool-using apps. Use it during design reviews, pre-release testing, and periodic security audits.

At a high level, prompt injection happens when untrusted content influences model behavior in ways you did not intend. In a basic chatbot, that may mean the model ignores its original instructions. In a RAG application, retrieved documents may contain hostile text like “ignore previous instructions” or hidden instructions embedded in markup, code blocks, or long-form content. In a tool-using agent, the risk is higher because malicious content may try to trigger actions, exfiltrate data, or manipulate downstream systems.

The most useful mental model is simple: treat every external input as untrusted, including user messages, retrieved documents, web pages, PDFs, emails, tickets, logs, database fields, and tool outputs. The model may read them all as if they are instructions unless you explicitly design around that behavior.

A strong prompt injection defense usually has four layers:

• Isolation: separate instructions from untrusted content as clearly as possible.
• Restriction: narrow what the model can do, especially with tool access and sensitive data.
• Verification: require checks before high-impact actions or structured outputs are accepted.
• Evaluation: test the system continuously with adversarial cases, not just happy-path prompts.

If you are building a broader AI stack, this checklist fits naturally with prompt testing and CI workflows. For teams formalizing that process, How to Build an LLM Evaluation Pipeline for CI/CD and Prompt Evaluation Metrics That Actually Matter in Production are useful next reads.

Checklist by scenario

This section breaks the checklist into the most common scenarios: plain RAG, tool-calling assistants, internal enterprise copilots, and multi-step agents. You do not need every control in every app, but you should be able to explain why each omitted control is unnecessary.

1. Core checklist for any LLM app

• Define trusted vs untrusted inputs. Document exactly which text sources count as instructions and which count as data. If everything reaches the model in one flattened prompt, your risk goes up immediately.
• Use explicit delimiters and role separation. System instructions, developer instructions, user requests, retrieved context, and tool outputs should be clearly segmented. This will not eliminate attacks, but it makes failure modes easier to reason about.
• Tell the model how to treat retrieved or external content. State plainly that external content may contain malicious or irrelevant instructions and must be treated as information, not authority.
• Minimize the model’s freedom. Prefer narrow tasks, bounded output formats, and schema validation over open-ended “do whatever is needed” prompts.
• Do not trust the model to self-police. A model saying “I detected prompt injection” is not the same as a defense. Put enforcement in application logic.
• Log prompt assembly and tool decisions. Keep enough telemetry to reconstruct what the model saw, what it selected, and why the app allowed the result.

2. RAG security checklist

For retrieval systems, your main concern is that hostile or low-quality documents can steer the model. A practical RAG security checklist should include the following:

• Sanitize retrieved content before inclusion. Strip or normalize hidden text, excessive markup, unusual delimiters, script-like fragments, and irrelevant prompt-like scaffolding where possible.
• Preserve source boundaries. Pass documents as separate items with identifiers instead of one merged blob. This makes source-aware reasoning and filtering easier.
• Show provenance to the model and the user. Source IDs, titles, trust tiers, timestamps, and repository labels can help both application logic and human reviewers.
• Use retrieval filters based on trust level. Separate public web content, internal knowledge base content, user-uploaded files, and machine-generated records. They should not all be treated equally.
• Constrain context windows. Large unfiltered context can hide malicious instructions among useful content. Retrieval quality often matters more than raw volume.
• Chunk documents with security in mind. Smaller, coherent chunks can reduce the chance that a single poisoned section contaminates an entire answer. For chunking tradeoffs, see RAG Chunking Strategies Compared.
• Consider index-time screening. Flag documents with instruction-like language, policy override patterns, or anomalous formatting before they enter the vector store.
• Test retrieval poisoning scenarios. Add adversarial documents that attempt to override rules, request secret disclosure, or manipulate citations.
• Require citation-grounded responses when appropriate. If the app must answer from sources, reject or downgrade outputs that make unsupported claims.

Architecture decisions also matter. Your choice of embeddings, chunking, and storage can change what gets surfaced and how often risky fragments appear. Related reading: Embedding Model Comparison for Semantic Search and RAG and Best Vector Databases for RAG.

3. Tool calling security checklist

If your assistant can call search, CRM, code execution, database, messaging, or file tools, tool calling security deserves its own review. Prompt injection becomes more dangerous when model output triggers actions.

• Use allowlisted tools only. The model should never invent arbitrary tools or endpoints.
• Give each tool the minimum privileges required. Read-only access is safer than write access. Scoped tokens are safer than broad credentials.
• Separate selection from execution. Let the model propose a tool call, but validate arguments and permissions in code before execution.
• Validate all tool inputs. Enforce schemas, type checks, length limits, field allowlists, and business-rule validation.
• Classify high-risk actions. Sending messages, writing records, making purchases, deleting data, changing permissions, and running code should require confirmation or an approval gate.
• Treat tool outputs as untrusted too. Search results, API responses, HTML, logs, and third-party plugin outputs can contain prompt injection payloads.
• Disable recursive authority. A tool result should not be able to redefine system rules or grant itself more permissions.
• Rate-limit and monitor tool use. Unexpected bursts of calls, repeated failed validations, or unusual argument patterns are worth alerting on.
• Design for safe failure. If validation is inconclusive, the application should decline the action rather than proceed optimistically.

If your team is exploring ecosystem-level tooling patterns, the Model Context Protocol Tools Directory for Developers is a useful reference point for thinking about tool surfaces and integrations.

4. Internal copilot checklist

Internal assistants often feel safer because the audience is employees, but they may have access to far more sensitive data. An internal app should still assume untrusted inputs.

• Segment data by role and sensitivity. The model should not retrieve or synthesize data a user is not allowed to access directly.
• Apply authorization before retrieval, not after answer generation. Post-hoc filtering is weaker than limiting the retrieval set upfront.
• Prevent secret leakage in context assembly. API keys, tokens, private notes, internal prompts, and hidden metadata should never enter the model context unless truly required.
• Review connectors carefully. Email, chat, wiki, ticketing, and document systems can all introduce hostile or accidental instructions.
• Use red-team prompts against realistic workflows. Test payroll, HR, support, finance, engineering, and admin use cases separately because the blast radius differs.

For domain-specific deployment choices, model selection still matters. If your use case overlaps with service workflows, How to Choose the Right LLM for Customer Support Automation can help frame capability and risk tradeoffs.

5. Multi-step agent checklist

Agents increase the number of decision points, which increases the number of places a prompt injection can succeed.

• Limit step count and tool depth. Infinite or loosely bounded loops make it easier for an injected instruction to persist.
• Persist only necessary memory. Do not carry forward every prior message or tool result if it no longer matters.
• Re-check policy at every action boundary. Passing validation once at the start is not enough.
• Keep a human in the loop for impactful workflows. This is especially important for external communication, data modification, and anything with compliance implications.
• Use deterministic checks around model decisions. For example, allowed domains, file types, command categories, and record scopes should be enforced outside the model.

What to double-check

This section is the fast audit pass. If you only have fifteen minutes before a review, check these items first.

• Your system prompt is not your security boundary. It is guidance, not a reliable enforcement mechanism.
• Retrieved content is not being appended as plain text without labels. If it is, the model may treat it like a fresh instruction source.
• Tool outputs are filtered and validated. Many teams sanitize user input but forget that external APIs can return hostile content too.
• Authorization happens before retrieval and before tool execution. Not after.
• Write actions require additional checks. Read actions and write actions should never share the same trust assumptions.
• Structured outputs are validated against a strict schema. This reduces accidental or malicious drift in downstream logic.
• You have adversarial test cases in your eval suite. If you only test helpful user behavior, your confidence is inflated.
• Logs capture enough context to investigate incidents. That includes retrieval results, prompt segments, selected tools, validation failures, and final actions.
• Fallback behavior is safe. If the model is uncertain, the app should ask for clarification, refuse, or route to review rather than improvise.

If you are still building your evaluation discipline, prompt injection tests belong alongside normal quality benchmarks. The goal is not only “does the answer look good?” but also “did the application stay within bounds under pressure?”

Common mistakes

These are the patterns that repeatedly weaken LLM prompt injection mitigation efforts in production systems.

Relying on a stronger warning message

Many teams respond to prompt injection by making the system prompt longer and firmer. Clear instructions help, but they are not enough by themselves. If your application still grants broad tool access or passes untrusted data directly into the context, the structural weakness remains.

Treating detection as prevention

Classifier prompts, heuristic filters, and model-based detectors can help prioritize review, but they will miss cases. Use them as signals, not as your only line of defense.

Giving the model excessive autonomy

“Figure out the best next step” sounds flexible, but it often collapses trust boundaries. The more action space you expose, the more validation and policy logic you need outside the model.

Merging all context into one prompt

Flattening system rules, user text, retrieved documents, and tool outputs into a single undifferentiated block makes it harder to enforce semantics. Preserve structure wherever possible.

Ignoring ingestion risk

Some prompt injection problems begin long before inference. Poisoned knowledge base entries, adversarial comments, and manipulated support tickets can all enter the corpus and later be retrieved as if trustworthy.

Skipping scenario-specific testing

A general prompt safety test is not enough. A document extraction assistant, a coding agent, and a customer support copilot have different tools, permissions, and failure costs. Tailor your tests to the workflow.

For teams comparing coding assistants or self-hosted setups, security review should be part of platform choice, not something added at the end. Depending on your environment, these related guides may help: AI Coding Assistant Comparison and Open Source LLMs for Self-Hosting.

When to revisit

This section is your action plan. Revisit this checklist whenever the model, workflow, or data surface changes.

At minimum, review your defenses in these situations:

• Before major planning cycles. If a new quarter or planning season introduces fresh integrations, new document sources, or broader internal rollout, review trust boundaries first.
• When workflows or tools change. Every new connector, plugin, write action, or approval path creates a new place for injected instructions to matter.
• When your retrieval corpus changes materially. New repositories, user-uploaded documents, public web search, and imported ticket histories all affect risk.
• When you switch models or prompting patterns. Different models may follow instructions, parse markup, or use tools differently.
• After incidents or near misses. Even a benign failure is a signal to update your tests and controls.
• Before expanding permissions. If the assistant is moving from read-only to write-capable behavior, do a dedicated security review.

A practical recurring workflow looks like this:

1. List new inputs, tools, and outputs added since the last review.
2. Classify each one as trusted, partially trusted, or untrusted.
3. Check whether any new source can inject instructions into the model context.
4. Review tool permissions and confirm least privilege still holds.
5. Run adversarial prompts and poisoned-document tests against real workflows.
6. Inspect logs for unsafe behavior, validation failures, and unexpected tool requests.
7. Update prompts, validators, and approval gates based on what you find.
8. Add the new failure cases to your permanent evaluation suite.

The goal is not to create a perfect static defense. The goal is to keep the application’s real behavior aligned with your intended security boundaries as the system evolves. In practice, that means prompt engineering, application logic, retrieval design, and evaluation all have to work together.

If you want this article to function as an internal operating checklist, copy the sections above into your release template and require an explicit yes, no, or not-applicable answer for each control. That small process change often does more for AI application security than another round of vague prompt tweaks.