Prompt Injection Prevention: A Security Checklist for LLM Applications

Overview

Prompt injection is an attempt to influence an LLM by placing instructions in content the application sends to the model. The content may be a direct user message, but it can also be indirect: a document, email, support ticket, web page, code comment, image, or search result may contain text designed to override the application's intended behavior.

The central security principle is simple: do not assume that instructions and data are safely separated because they appear in different sections of a prompt. A model may interpret both as language. Your application therefore needs controls outside the prompt itself, including least-privilege tool access, authorization checks, data filtering, output validation, logging, and human approval for consequential actions.

Prompt engineering remains useful for stating priorities clearly, but a prompt is not an access-control system. A well-written system instruction cannot replace server-side authorization or prevent a tool from receiving excessive permissions. For broader workflow patterns, see the guide to AI agent workflow design.

A practical threat model

Before choosing mitigations, map the workflow:

• Assets: Identify secrets, personal data, internal documents, credentials, source code, financial records, and external systems the application can access.
• Inputs: List every source of text, images, files, retrieved passages, tool output, and user-provided data.
• Actors: Consider ordinary users, malicious users, compromised accounts, untrusted content authors, and third-party services.
• Actions: Record what the model can read, write, send, purchase, delete, publish, or execute.
• Failure impact: Rank the consequences of data disclosure, unauthorized action, incorrect output, service abuse, and loss of auditability.

Checklist by scenario

1. User-facing chat

• Define which requests the assistant is meant to handle and which it must refuse or route elsewhere.
• Keep authentication and authorization outside the model. The model may request an operation, but application code should decide whether the user is allowed to perform it.
• Do not place secrets, unrestricted internal instructions, or unnecessary customer records in the context.
• Set clear limits for account changes, refunds, deletion, messaging, and other consequential actions.
• Return errors and refusal messages that do not disclose hidden prompts, credentials, internal identifiers, or sensitive records.
• Rate-limit requests and monitor repeated attempts to extract instructions or sensitive data.

2. Retrieval-augmented generation

RAG creates an indirect prompt injection surface because retrieved content can contain instructions that were never approved by the application owner. A document should be treated as evidence or data, not as an authority that can redefine the workflow.

• Classify sources by trust level and preserve source metadata through retrieval and generation.
• Tell the model to distinguish quoted content from application instructions, while recognizing that this instruction alone is not a complete defense.
• Restrict retrieval to documents the requesting user is authorized to view.
• Filter or quarantine content that contains suspicious requests to reveal secrets, ignore constraints, or invoke tools.
• Require citations or source references where appropriate, then validate that cited material actually supports the answer.
• Test poisoned documents, conflicting passages, hidden text, and instructions embedded in markup or code.

Use a documented ingestion policy for files and external sources. The RAG tutorial and RAG application architecture guide provide useful context for separating retrieval, generation, and application controls.

3. Tool-using agents

• Give each tool the smallest useful permission set. Prefer read-only access when writing is not required.
• Use explicit schemas with typed, constrained parameters rather than allowing free-form commands.
• Validate tool arguments in application code, including destination, amount, record identifier, file path, and query scope.
• Require confirmation before sending messages, changing records, making purchases, deleting data, deploying code, or accessing sensitive systems.
• Use separate credentials and environments for development, testing, and production.
• Set time, call-count, budget, and recursion limits so a manipulated agent cannot run indefinitely.
• Log the user request, selected tool, validated arguments, authorization decision, result, and approval event.

4. Browsing, email, and external content

• Assume web pages, emails, attachments, and search results may contain indirect injections.
• Separate browsing from execution. Reading a page should not automatically grant permission to follow its instructions.
• Strip or isolate unnecessary scripts, hidden text, metadata, and markup before passing content to the model.
• Prevent the assistant from forwarding sensitive context to arbitrary URLs or email recipients.
• Require a user or service-level approval before external communication or file upload.

5. Structured output and code generation

• Constrain responses to a schema and reject malformed or unexpected fields.
• Validate strings, URLs, identifiers, file paths, and generated queries before use.
• Never execute generated code, SQL, shell commands, or configuration without isolation, review, and allowlisting.
• Escape output for its destination, such as HTML, SQL, Markdown, logs, or a shell environment.
• Use a fallback path when the output cannot be parsed or fails policy checks.

Attack-and-mitigation table

What to double-check

Use this review before releasing a new LLM feature or changing an existing workflow:

1. Instruction hierarchy: Can the application clearly distinguish policy, task instructions, user content, retrieved content, and tool output? Treat the distinction as a design aid, not a security boundary.
2. Data minimization: Does the model receive only the fields needed for the current task? Redact secrets and unnecessary personal or internal data before inference.
3. Tool permissions: Can a successful injection reach a high-impact tool? Reduce permissions, isolate credentials, and add approval gates.
4. Authorization: Does the backend independently verify the user, resource, operation, and target? Never rely on the model to make the final access decision.
5. Output handling: Are responses parsed, validated, escaped, and safely passed to downstream systems?
6. Observability: Can you reconstruct what content entered the context, what the model proposed, which controls ran, and what action occurred?
7. Evaluation: Are adversarial cases included in automated prompt testing and regression checks? The prompt testing and evaluation framework can help organize this work.
8. Recovery: Can operators revoke credentials, disable a tool, quarantine a data source, and roll back an action quickly?

Maintain separate tests for direct injection, indirect injection, data exfiltration, tool misuse, instruction leakage, and denial-of-service behavior. A test should specify the input, expected safe behavior, unacceptable behavior, severity, and whether a human review is required. Reuse the same cases after prompt, model, retrieval, tool, or provider changes.

Common mistakes

• Relying on a stronger prompt alone: Prompt wording can reduce ambiguity but cannot enforce permissions or guarantee that untrusted content will be ignored.
• Putting secrets in the context: If the model does not need a credential or private instruction to complete the task, do not send it.
• Giving one broad tool access to everything: Split capabilities into narrow tools with explicit inputs and independent authorization.
• Trusting retrieved documents: Relevance does not imply trust. Content can be accurate, outdated, malicious, or merely written with a different audience in mind.
• Checking only the final answer: A harmless-looking answer may conceal an unsafe tool call or an attempted external action. Inspect intermediate decisions and tool arguments.
• Logging sensitive prompts indiscriminately: Logs are valuable for investigation but can become a second data exposure. Redact, restrict, and retain them according to your security requirements.
• Skipping human approval because the task seems routine: Define approval thresholds based on impact, reversibility, destination, and data sensitivity.

When to revisit

Review this LLM security checklist at least quarterly as part of normal application security work, and sooner whenever the workflow changes. Trigger a focused review when you add a model, prompt, tool, data source, browser capability, plugin, integration, or production permission. Also revisit it after a security incident, a failed red-team test, a change to authentication, or a major change in provider safeguards or framework guidance.

Before each seasonal planning or release cycle, confirm that the threat model still matches the application. Ask:

• What new data can the system read?
• What new actions can it take?
• Which users or services gained access?
• Did prompts, retrieval rules, schemas, or approval thresholds change?
• Do regression tests cover the new paths?
• Can the team disable unsafe tools and investigate an incident?

Record the review date, application version, tested attack cases, open risks, owners, and planned mitigations. Security improves when this checklist is part of the AI development workflow rather than a one-time launch document. For related implementation work, consult the AI API integration guide and the resources on building a prompt testing workflow.