First reported medium.com
Research · latest
First reported arxiv.org
Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems
Researchers at Tampere University present Trustworthy RAG, an Evaluation Agent middleware that detects knowledge poisoning and misinformation in Retrieval-Augmented Generation systems by combining Natural Language Inference factual verification, a five-signal poison detector, and a Trust Index scoring formula. On TruthfulQA with Llama 3.3 70B the agent reaches 91% accuracy and 100% recall on instruction injection, though subtle in-place entity swaps remain hard to detect; the authors release the approach, an attack generator, and experimental artifacts at github.com/GPT-Laboratory/TrustworthyRAG. Details →First reported arxiv.org
InjecMEM: Memory Injection Attack on LLM Agent Memory Systems
InjecMEM, described in a paper by researchers from Shanghai Jiao Tong University and Ant Group, is a memory-injection attack that plants hidden instructions in an LLM agent's persistent memory with a single interaction and no read/edit access to the memory store. Using a retriever-agnostic anchor plus a gradient-optimized adversarial command, it steers later responses on a target topic, achieving up to 35.4% retrieval success and 76.6% attack success on MemoryOS while leaving non-target queries unaffected; code is published on GitHub. Details →First reported escape.tech
AI vs AI: How Cascade exploited an AI agent in production
Escape's AI pentesting engine, Cascade, bypassed a production AI agent's prompt-injection guardrail on its second attempt by reframing the same request as an innocuous research/documentation query, causing the agent to disclose its full system prompt, tool list, tool-calling rules, output formatting, and session identifiers. The bypass relied on social-engineering-style pretexting rather than a technical exploit, with Cascade autonomously rewording its payload after reading the initial refusal. Details →First reported · updated · 2 reports aclanthology.org
MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Knowledge Poisoning Attacks
MM-PoisonRAG is a research paper presenting local and global knowledge poisoning attacks that disrupt multimodal retrieval-augmented generation (RAG) in multimodal large language models (MLLMs). The work demonstrates how MLLMs' reliance on retrieval exposes them to poisoned knowledge injected into the retrieval corpus, manipulating generated outputs. Details →First reported arxiv.org
MAPLE-Guard: Memory-Aware Link EnforcementAgainst Memory-Link Poisoning in Multi-Agent Systems
MAPLE-Guard is a defense presented in an arXiv paper against memory-link poisoning in LLM-based multi-agent systems, where a single poisoned memory write can be retrieved, promoted into shared memory, and reused by agents that never saw the original attack. The guard places gates at write, retrieval, promotion, and cross-agent reuse, reportedly lowering attack success rate from 38.2% to 0.9% on LongMemEval and 34.7% to 0.2% on AppWorld, with code released on GitHub. Details →First reported arxiv.org
The Injection Paradox: Brand-Level Suppression in Safety-Trained LLM Recommendations via RAG Context Injection
An ICML 2026 workshop paper by Hyunseok Paeng, "The Injection Paradox," reports a reproducible failure mode in RAG-based LLM recommendation where prompt injections embedded in retrieved documents backfire, suppressing the injected brand below baseline in safety-trained Claude models (Claude Opus 4.6 dropped the target brand from a 54% baseline to zero top-2 recommendations). The suppression propagates to unmodified documents of the same brand, while GPT models showed the opposite effect, raising a reverse-attack scenario where an adversary poisons a competitor's documents to suppress that competitor's brand. Details →First reported arxiv.org
Hidden in Memory: Sleeper Memory Poisoning in LLM Agents
The arXiv paper "Hidden in Memory: Sleeper Memory Poisoning in LLM Agents" (2605.15338), explained in a companion Medium write-up, demonstrates a delayed attack in which adversarial content in a document, webpage, or repository causes a stateful LLM assistant to store a fabricated memory about the user that re-emerges across future sessions. Across tested assistants, poisoned memories were written up to 99.8% of the time on GPT-5.5 and 95% on Kimi-K2.6, and among successful retrievals they steered attacker-intended agentic actions in 60–89% of evaluations; the authors release code on GitHub. Details →First reported offensai.com
A Security Analysis of Amazon S3 Vectors and Its Use in LLM Retrieval Pipelines
OFFENSAI and TUCN researchers analyzed Amazon S3 Vectors, AWS's native vector store for RAG pipelines, and demonstrated that its metadata is completely unvalidated on write, letting anyone with s3vectors:PutVectors forge chunk text, spoof citation URLs, rank-flood indexes, or overwrite legitimate chunks. In testing, a single planted vector drove a clinical RAG assistant to recommend a dangerous drug dose while citing an authentic PDF, and a poisoned chunk led to command execution in a tool-enabled agent's sandbox, with CloudTrail unable to reconstruct what changed. Details →First reported arxiv.org
Defending Retrieval-Augmented Intrusion Detection Against Knowledge Poisoning and Prompt Injection
The paper "Defending Retrieval-Augmented Intrusion Detection Against Knowledge Poisoning and Prompt Injection" presents RAG-IDS, a three-tier multi-agent intrusion detection framework with a retrieval-boundary defense combining soft trust scoring, label-embedding consistency checking (LECC), and prompt sanitization. Experiments on CIC-UNSW-NB15 show recovery under knowledge poisoning and reduced label-flip success under prompt injection when using multi-document retrieval. Details →First reported · updated · 2 reports acm.org
When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapse | Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval
A SIGIR paper presents SilentRetrieval, a two-stage RAG data-poisoning attack that uses Coordinated Beam Search and Context-Adaptive Trigger Generation to craft fluent, low-perplexity adversarial documents that remain retrievable and hijack answers, achieving up to 84.6% HR@10 and 57.5% ASR-LLM on Natural Questions and MS MARCO, and retaining 74.2% HR@10 at a 0.016% poisoning ratio on a 21M-passage corpus. The work also evaluates combined retrieval- and generation-side defenses (reducing ASR-LLM to ~21-25%) and proposes detection via document-level attention collapse. Details →First reported arxiv.org
When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapse
Researchers from University of Technology Sydney and collaborators present D-SCAN (Document-level Signal Collapse Analysis), a lightweight framework that detects RAG poisoning attacks by monitoring an internal signature they call 'Attention Collapse'—where a generator's attention entropy decreases and concentrates on injected adversarial documents. The paper shows that output-side signals like perplexity fail because poisoned outputs can exhibit lower perplexity (false confidence), and that D-SCAN can detect attacks even when they do not alter the final answer, with code released on GitHub. Details →First reported arxiv.org
When Safety Becomes a Vulnerability: Exploiting LLM Alignment Homogeneity for Transferable Blocking in RAG
Researchers present TabooRAG, a blocking attack against Retrieval-Augmented Generation systems that exploits what they call 'alignment homogeneity'—shared refusal criteria across safety-aligned LLMs—to craft poisoned documents that cause models to refuse benign queries. Rather than injecting instructions, TabooRAG builds query-relevant risk context in a surrogate environment and transfers it to unknown target systems, achieving state-of-the-art attack success across nine LLMs and three datasets with a 67.3% relative gain over baselines even after prompt-injection filtering. Details →First reported icml.cc
ICML Poster Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractors
"Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractors," an ICML 2026 poster by Yen-Shan Chen and colleagues, presents a RAG data-poisoning technique that uses transferable attention-steering "attractors" to manipulate retrieval-augmented generation systems. The abstract is truncated, but the work is framed as a scalable, transferable poisoning attack against RAG pipelines. Details →First reported arxiv.org
Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications
The paper 'Here Comes The AI Worm' by Cohen, Bitton, and Nassi demonstrates Morris-II, a zero-click self-replicating adversarial prompt that triggers cascading indirect prompt injections across RAG-based GenAI applications such as email assistants, forcing each to exfiltrate confidential data and poison the RAG of additional applications. The authors evaluate propagation across context sizes, embedding algorithms, and hop counts, and introduce a guardrail called Virtual Donkey achieving a 1.0 true-positive rate at a 0.015 false-positive rate. Details →First reported darkreading.com
AI Harnesses Burst With Potential Exploit Opps
Researchers at AI penetration-testing firm Novee Security demonstrated that trust misalignments between the components of AI 'harnesses' (the software frameworks providing tools, memory, and guardrails around LLMs) can be exploited, including using Google's AI agent to execute a supply-chain attack and write to its own GitHub repository, per Dark Reading. The team also reported finding exploitable trust issues in Anthropic's and OpenAI's AI agents. Details →First reported · updated · 2 reports arxiv.org
DisarmRAG: Stealthy Retriever-Centric Poisoning to Disable Self-Correction in Retrieval-Augmented Generation (Extended Version)
DisarmRAG is a research attack framework that poisons the retriever component of Retrieval-Augmented Generation systems—rather than only the knowledge base—to inject anti-self-correction instructions into the LLM context, suppressing models' self-correction ability and forcing attacker-chosen outputs. Using iterative co-optimization and a contrastive-learning-based stealthy model-editing technique, the authors report success rates exceeding 90% across six LLMs and three QA benchmarks while evading detection defenses. Details →First reported arxiv.org
Forensic Trajectory Signatures for Agent Memory Poisoning Detection
The paper 'Forensic Trajectory Signatures for Agent Memory Poisoning Detection' by Jun Wen Leong identifies a behavioral invariant in LLM agents under persistent memory poisoning: successful attacks require calling memory_recall_fact before email_send_email. A rule and Random Forest classifier over trajectory features achieve high AUC (up to 0.99) across 9 models, but a preregistered follow-up (N=4,360, 13 models) reveals benign memory-grounded sends produce the same signature, causing high false positives; the recall_before_send pattern is an attack precondition rather than a maliciousness predicate, requiring recipient-metadata gating to restore separation. Details →First reported arxiv.org
CrackedPDFs: A Controlled Benchmark for Hidden Prompt Injection in PDFs
CrackedPDFs is a controlled benchmark of 29,322 generated PDFs (9,774 injected, 19,548 benign/confounder) for studying hidden prompt injection in documents that LLM pipelines flatten before guardrails inspect them. The authors evaluate PromptGuard, rule baselines, structural-only learned models, and a document-aware hybrid detector, which reaches 0.960 F1 with 0.998 ROC-AUC on a held-out set while noting PromptGuard's low recall on extracted text and the limits of real-world generalization. Details →First reported arxiv.org
KidnapRAG: A Black-Box Attack for Hijacking Reasoning in Agentic Retrieval-Augmented Generation Systems
KidnapRAG is a research paper presenting a black-box poisoning attack against Agentic RAG systems in which the attacker only publishes externally retrievable poisoned documents. The method uses three role-specific documents (Bait, Chain-Link, and Mal-Ins) to hijack an agent's multi-step reasoning chain, outperforming existing poisoning baselines across multiple frameworks, LLM backbones, and benchmarks; code is released on GitHub. Details →First reported arxiv.org
The Injection Paradox: Brand-Level Suppression in Safety-Trained LLM Recommendations via RAG Context Injection
The paper 'The Injection Paradox' by Hyunseok Paeng documents a reproducible failure mode in RAG-based LLM recommendation systems where indirect prompt injections embedded in retrieved documents backfire, suppressing the injected brand below its injection-free baseline. In safety-trained Claude models (e.g., Claude Opus 4.6), a single injected document drops the target brand from a 54% baseline to zero top-2 recommendations and propagates suppression to the brand's uninjected documents, while GPT models instead show increased recommendations. The authors note this enables a reverse-attack scenario where an adversary injects a competitor's documents to suppress that competitor, and release code, prompts, and results. Details →First reported aclanthology.org
Data Exfiltration via Backdoored Tool Use
Researchers present Back-Reveal, a data exfiltration attack that embeds semantic triggers into fine-tuned tool-use LLM agents; when triggered, the backdoored agent invokes memory-access tool calls to retrieve stored user context and covertly transmits it to an attacker-controlled server disguised as legitimate retrieval/search API requests. The paper shows multi-turn interaction amplifies leakage, as attacker-controlled retrieval responses steer subsequent agent behavior to induce cumulative disclosure, and evaluates defenses like NeMo Guardrails and LLM Guard as targets. Details →First reported akamai.com
AI Reconnaissance: The Missing Layer in Chatbot Security
Akamai's AI security research team describes "AI reconnaissance," a technique in which deployed AI assistants block direct requests for sensitive data but readily leak operational context — capabilities, boundaries, knowledge-base access, and tool/action surface — when asked simple, benign-looking questions. This leaked context lets attackers craft targeted, application-specific prompt injections that bypass model-level safety, and the authors argue runtime guardrails must detect this early probing stage of the attack kill chain. Details →First reported arxiv.org
A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots
A research paper by Gulshan Saleem and colleagues presents a three-layer middleware framework to defend RAG-based chatbots against both direct and indirect prompt injection, combining input screening, a provenance-based instruction hierarchy during context assembly, and output auditing. Evaluated on 5,080 samples across GPT-4o, Llama 3, and Mistral 7B, the framework reduced Attack Success Rate from 71.4% to 11.3% while maintaining a 4.8% false positive rate and 61.2 ms median latency overhead. Details →First reported koreshield.ai
Reproducing an Indirect Prompt Injection Against a RAG Pipeline
A Koreshield write-up reproduces an indirect prompt injection against a RAG customer-support pipeline, where a poisoned knowledge-base document (doc-002) contains a bracketed fake 'system note' instructing the model to dump all retrieved documents and exfiltrate internal tokens with an 'EXFIL::' marker while hiding the action. The author provides runnable code, a public GitHub harness (UncleTeslim/prompt-injection-repro), and real model logs demonstrating the attack against a local Llama3-8B model. Details →First reported lateos.ai
GPT-5 Nano IPI Assessment — LLM Vulnerability Research
A black-box prompt injection susceptibility assessment of GPT-5 Nano using the IPI Taxonomy v0.13 across 201 analyzed test cases, reporting a 38.3% overall susceptibility rate. The model was fully resistant to surface-level attacks (CSS concealment, HTML cloaking, SEO phishing, RAG corpus poisoning) but highly vulnerable to recursive instruction framing (100%) and MCP tool description poisoning (80%). Details →How the wire is made
Poll & cluster
Internet is crawled for AI security news and near-duplicate coverage is embedded and grouped into durable items.
Curate
AI Agent filters for agentic-AI relevance, classifies and tags each item, scores severity for threats, and writes the summary.
Every item here is one machine-curated intelligence object, not a headline.
Read the wire for free. There is a small charge to ask the index questions.
The wire, open
The complete curated feed, no key required.
- GET /feed.xml — RSS 2.0, every item
- GET /api/items — read-only
The vector desk
Query the index by meaning, not just keyword.
- GET /api/items?tags=&minSeverity=&itemType=
- GET /api/search?q= — keyword
- GET /api/semantic?q= — vector