Prompt Injection: How to Protect AI Agents and LLM Apps

Summary

This material is explanatory and defensive guidance on prompt injection, the class of attack in which content processed by an AI system influences the model to follow instructions that conflict with the application's intended behavior or the user's actual request. It frames the problem as an instruction-versus-data trust boundary: a language model processes trusted instructions and untrusted data through the same context window, so untrusted content can contain instruction-like text without gaining real authority. OWASP ranks prompt injection as LLM01, the top risk in its 2025 Top 10 for LLM applications, and the guidance stresses that no single system prompt makes an agent injection-proof — only layered, risk-reducing defenses.^[0][138]

The guidance and its cited sources anchor the discussion in the attack's history and its evolution. Prompt injection was publicly demonstrated against GPT-3 and named in September 2022, and OpenAI now characterizes it as a social-engineering attack specific to conversational AI in which a third party injects malicious instructions into the conversation context. OpenAI further reports that the most effective real-world attacks increasingly resemble social engineering rather than crude prompt overrides — for example a 2025 externally reported (ShadowLeak) attack on ChatGPT deep research that succeeded 50% of the time — and warns that intermediary 'AI firewalling' input classifiers often fail against such fully developed attacks. All sources emphasize defensive controls rather than a specific actively exploited campaign or named threat actor.^{[117][22][23]}

Recent academic work extends the threat beyond chatbots and agents. The LogInject study demonstrates 'passive prompt injection' against LLM-based security-operations log analysis with attack success rates as high as 88.2%, using a 'Context Stitching' technique that fragments payloads across log entries to evade stateless filters, while layered defenses cut attacks by 90.4% but left an 8.4% residual. Other 2026 research shows prompt-injection techniques that manipulate and exfiltrate models' own reasoning traces. Together these reinforce that the underlying confused-deputy problem is structural and recurring across new deployment patterns.^[138][29]

How it works

The weakness is structural rather than a patchable bug. In traditional software, program logic and data are separated, but in an LLM the instructions, user input, documents, tool results, and external text are all represented as tokens inside one model context. Applications can label system instructions, user messages, and retrieved content differently, but those labels do not create a reliable security boundary — once untrusted natural-language content reaches the context, malicious instructions inside it can compete with and override the developer's or user's intent.^[0]

The guidance distinguishes direct injection (the user or third party crafts input that redirects the model) from indirect injection (an attacker plants instructions in content the model will later ingest — web pages, emails, documents, RAG chunks, tool outputs, MCP resources, or images with text). RAG does not eliminate the risk because retrieved documents can themselves carry malicious instructions, and large context windows can increase the volume of untrusted material reaching the model. OpenAI's source-sink framing captures the high-impact case: an attacker needs a manipulable source and a dangerous sink.^[0]

The LogInject research generalizes the confused-deputy mechanism to security log pipelines: adversaries embed prompt-injection payloads in log-generating fields (achieving 'passive prompt injection') that persist in storage and execute when a SOC analyst later queries the LLM for summarization or triage. A 'Context Stitching' variant fragments a payload across multiple log entries so no single fragment trips a stateless filter, yet the full instruction assembles inside the model's long-context window, producing a 76.4% success rate.^[138]

A further variant manipulates a model's own reasoning: an attacker induces a model to contemplate data exfiltration within its chain-of-thought and then replays that (encrypted) reasoning trace into another model, exploiting the tendency of models to treat their own reasoning traces as trusted.^[29]

Key takeaways

• Prompt injection cannot be fully patched because a model cannot structurally separate instructions from data; there is no single prompt that makes an agent 'prompt-injection-proof,' and system prompts should not be treated as a security boundary or a place to store secrets.^[0]
• Risk scales with what the model is connected to: manipulated model behavior in an agent with tools, private data, and external communication can lead to data access, exfiltration, or unauthorized actions, so an attacker needs both a manipulable source and a dangerous sink for high-impact exploitation.^[0][22]
• The attack pattern keeps resurfacing in new contexts — from the 2022 GPT-3 demonstrations, to OpenAI's observation that real-world attacks now resemble social engineering (e.g. the ShadowLeak deep-research attack), to high-success passive injection against SOC log-analysis pipelines and manipulation of models' own reasoning traces — underscoring an evolving, industry-wide confused-deputy problem that OWASP ranks as the top LLM risk.^{[117][23][138][29]}

Defensive actions

• Enforce least privilege, limiting an agent's access to only the data and tools it needs to complete a task, and prefer read-only access over write, send, delete, purchase, deploy, or administrative capabilities.: The guidance and OpenAI both call limiting access the most effective real-world control: an agent that cannot reach sensitive data or take irreversible actions cannot be made to leak or destroy them, reducing the blast radius of any successful injection.^[0][22]
• Require deterministic runtime authorization and explicit user approval for consequential actions, and never let retrieved or external content grant itself new permissions.: Human-in-the-loop confirmation before consequential actions and refusing content-driven privilege escalation are often the last controls that stop a successful injection from becoming real damage; OpenAI provides confirmations prior to consequential actions as a user control.^[0][22]
• Apply layered defenses combining input filtering, prompt hardening, and output validation, and validate tool arguments and model outputs before software executes or trusts them, with sandboxing and isolation for generated code or actions.: The LogInject study found combining input filtering, prompt hardening, and output validation reduced attacks by 90.4% though an 8.4% residual persisted, so defense-in-depth plus human oversight is required rather than reliance on any single filter.^[138][0]
• Do not rely on malicious-string detection or intermediary 'AI firewalling' alone, and test prompt-injection resistance across complete agent trajectories rather than only final text responses.: OpenAI reports that input-classification firewalls frequently fail against fully developed social-engineering injections because detection becomes as hard as detecting a lie without context, and the guidance stresses that simple blacklists such as blocking 'ignore previous instructions' are insufficient against adaptive attacks.^[23][0]

Changelog

• Added OpenAI's 'Designing AI agents to resist prompt injection' analysis, which reframes the most effective real-world attacks as social engineering rather than simple prompt overrides, cites a 2025 externally reported (ShadowLeak) attack on ChatGPT deep research that succeeded 50% of the time, and warns that intermediary 'AI firewalling' input classifiers often fail against fully developed attacks.^[23]