Publication date unknown · Discovered arxiv.org
Topic · Latest 50 of 398 active items
AI Security Research
Research artifacts and technical findings that clarify how AI systems fail, can be evaluated, or can be defended.
Matching wire items
Publication date unknown · Discovered ieee.org
Generative AI Security and Privacy Threats: A Systematic Evaluation and Defense Framework
Meta-DistillDef is a proposed generative-AI defense algorithm combining meta-learning, adversarial distillation, and multimodal fusion for adaptive threat detection and privacy protection. The paper reports evaluations on GLUE, CIFAR-10, and MSCOCO, claiming detection accuracy up to 98.3% and a privacy leakage rate as low as 3.1%. Details →Publication date unknown · Discovered bitsight.com
AI in Cybercrime: What 16 Underground Forums Reveal
Bitsight analyzed thousands of posts across 16 underground communities from February through August 2026, tracking discussion of jailbreaks, stolen API keys, AI models, agents, and MCP. The researchers report increasing interest in targeting AI systems and a shift toward autonomous agents, open-weight models, and productized attack infrastructure, while acknowledging the limitations of keyword-based measurement. Details →First reported · Discovered arxiv.org
Where AI Vulnerability Discovery Harnesses Go Quiet: Structural False Negatives and Hidden Vulnerabilities
Gadi Evron's study of OpenAnt, raptor, and 10 additional LLM-based vulnerability discovery harnesses identifies 13 ways code selection, filtering, and findings handling can suppress real vulnerabilities independently of model capability. Using the harnesses' own code, the study demonstrates a file being excluded before model analysis by six harnesses, five without notice, and also examines content-steered verdicts and prompt injection. Details →Publication date unknown · Discovered arxiv.org
CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
CyberArena is a dynamic benchmark in which AI agents simultaneously attack and defend changing services in isolated sandboxes, testing vulnerability discovery, exploitation, and patching. The paper describes automated environment generation and an evaluation involving 12 frontier LLM agents, 40 challenges, and 466 matches. Details →Publication date unknown · Discovered arxiv.org
Persistent Billable State: Denial-of-Wallet Attacks and Defenses in Tool-Calling LLM Agents
Persistent Billable State studies denial-of-wallet attacks in which malicious or compromised tool outputs remain in agent history and incur repeated input-processing charges. Across 243 executions spanning six model families, the authors report maximum cumulative session input reaching 14,293× the corresponding first-call input. The study evaluates history transformation and host-side limits on prompt size, context growth, recursion, and cumulative spending. Details →Publication date unknown · Discovered arxiv.org
Poisoning the Neighborhood: A Topology-Aware Topic-Level Attack on Large-Scale HNSW-based RAG Systems
Poisoning the Neighborhood presents a topology-aware poisoning attack against HNSW-based RAG systems that generalizes to unseen topic queries. Poisoned documents form a Hub–Satellite structure to improve retrieval, then couple a target entity with an attacker-chosen claim to influence generated answers. On a two-million-document Wikipedia corpus, the authors report over 72% retrieval success and 31.29% attack success. Details →Publication date unknown · Discovered arxiv.org
SoK: PIAtlas: Decomposing the Unbounded Prompt Injection Attack Space
PIAtlas introduces a prompt-injection taxonomy organized around six instruction-subversion mechanisms. The researchers release PICorpus and map thousands of attack records using a human-verified protocol, finding that attacks commonly combine techniques and that widely used agent benchmarks underrepresent mechanisms favored by adaptive attackers. Details →First reported · Discovered snyk.io
AI Model Risk Depends on What the Model Can Reach
Snyk research matched model identifiers in 1,693 enterprise environments to adversarial tests covering 29 attacker goals, finding that 49.2% referenced a model with at least one critical score. PII extraction and command execution accounted for critical findings, but the study does not establish production exposure or successful attacks against those applications. Only 37.5% of detected model occurrences had matched scores. Details →First reported · Discovered crowdstrike.com
Unknown Threat Actor Uses AI-Driven ARTEX to Target South Korean Finance
Flare reports a 745% increase in cybercrime-channel mentions of model abliteration from January to October 2026, based on analysis spanning more than 100,000 Telegram channels and 100 dark-web forums. The study identifies a small model-access marketplace dominated by apparent API resale rather than verified weight-level modification, and cites CrowdStrike’s investigation of ARTEX-assisted intrusions and data theft at South Korean financial organizations. Details →First reported · Discovered securityboulevard.com
AI Models are Fueling a New Wave of Cybercrime: A Quantitative Analysis
Flare reports analyzing more than 100,000 Telegram channels and 100 dark web forums to examine cybercriminal discussion, sale, and use of open-weight models with removed guardrails. Supporting CrowdStrike reporting documents ARTEX and LLM use in a campaign against South Korean financial organizations that resulted in data exfiltration. Details →Publication date unknown · Discovered arxiv.org
SEAL: Towards Persistent Tool Poisoning Attack against LLM Agents via Prefix-Eliciting Trigger
SEAL is a research method for persistent tool poisoning that optimizes triggers to elicit attack-relevant reasoning prefixes, then aggregates them into a universal trigger across agent contexts and execution rounds. Experiments across seven multi-round scenarios and multiple models report attack effectiveness, cross-model transferability, and resistance to tested defenses. Details →First reported loomcycle.dev
Decision model as prompt-injection judge: 40 of 40 on 78 texts
Loomcycle evaluates single-pass, typed-output decision models as replacements for a chat-based prompt-injection judge, using 40 injection examples and 38 clean hard negatives. The write-up claims 40-of-40 injection detection and says only one of four question formulations worked, although the provided excerpt does not include detailed results. Details →Publication date unknown · Discovered arxiv.org
Beyond Maliciousness: Static and Runtime Attack-Surface Profiling of Agent Skills
“Beyond Maliciousness” studies 6,309 agent-skill marketplace listings using static analysis, dependency provenance checks, and instrumented sandbox execution. Seven in ten listings carry evidence of at least one elevated capability, while 59% of listings declaring permissions understate their bundles’ capabilities. The study finds that permission declarations remove confirmation prompts rather than restrict access. Details →Publication date unknown · Discovered arxiv.org
BRANCH: Bypassing Multi-Scanner AI Guardrails
BRANCH is a research methodology that uses branching tree search and scanner-specific adversarial perturbations to bypass multi-scanner AI guardrails while preserving prompt meaning. The authors report 100% attack success across six guardrail systems in 120 scenarios, with 72% fewer queries than comparison techniques, and demonstrate transfer to 29 unseen guardrails, including eight commercial black-box systems. Details →Publication date unknown · Discovered arxiv.org
TRAP: Tree-Guided Reinforcement-Augmented Probing for Automated LLM Jailbreaking
TRAP is an automated black-box LLM jailbreaking method that combines a hierarchical strategy tree, reinforcement learning, and retrieval of successful attack examples to adapt attacks across objectives. Experiments on AdvBench report that TRAP outperformed comparison methods on six of seven aligned LLMs, with attack success rates of 45.6%–97.9% on those six models under the study's evaluation settings. Details →Publication date unknown · Discovered arxiv.org
TRACER: Trace-Regularized Training Against Indirect Prompt Injection
TRACER is a channel-aware adversarial training method for defending LLMs against indirect prompt injection in retrieved content, tool outputs, and agent messages. The study reports that identical attacks averaged 18% success as follow-up messages versus 53% in retrieved context; on GSM8K with Qwen2.5-3B-Instruct, TRACER reduced RAG-framed attack success from 65.5% to 6.1% while preserving 78–82% clean accuracy. Details →First reported · Discovered goteleport.com
Don't Trust the Sandbox: 5 Security Lessons from an AI Agent Swarm
Teleport's Ben Arent describes a capture-the-flag experiment involving five sandboxed agents, each holding an answer fragment and required to collaborate. The write-up reports coordination through shared state, custom protocols, and self-assigned identities, arguing for infrastructure-enforced capability and identity controls rather than reliance on sandboxing alone. Details →Publication date unknown · Discovered kairos-experiment.com
Three checks your memory evaluation should pass — Kairos
Kairos tested counterfeit and scrambled agent-memory documents against an LLM-based evaluation, finding equivalent identity-marker scores for fabricated and genuine histories across seven test items. Memory-reference results were inconclusive, and rescoring identical responses in different pairings revealed judge-score drift. Details →Publication date unknown · Discovered arxiv.org
Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!
Tsinghua-led researchers demonstrated that a malicious open-weight LLM provider can backdoor a checkpoint to extract downstream fine-tuning prompts using only black-box access to the resulting model. Experiments recovered up to 76.3% of 5,000 private queries, rising to 94.9% with prior knowledge of query beginnings. The attack depends on downstream training computing loss on query tokens; the report describes controlled experiments, not observed exploitation. Details →Publication date unknown · Discovered arxiv.org
Understanding and Enhancing Backdoor Persistency in LLM Agent Post-Training
The PERSISTBD study demonstrates that backdoors in software-engineering agent models can survive benign supervised fine-tuning (SFT) and subsequent reinforcement learning (RL). On Qwen2.5-Coder-7B, the proposed refinement method increased attack success from 20% to 74% after SFT and from 20% to 76% after SFT–RL, while maintaining comparable benign task performance. Details →First reported darkreading.com
AI Takes Control of Robotic Arm in 16 Minutes Flat
Booz Allen's laboratory tests found that commercial AI models could identify misconfigured OT devices, traverse multiple network layers, and trigger physical actions. In one test, a model took control of and moved a robotic arm within 16 minutes; the report describes controlled testing rather than a real-world intrusion. Details →First reported · Discovered · 4 reports anthropic.com
AI models may be capable of self-replicating without access to a datacenter
Anthropic’s Frontier Red Team demonstrated Gemma4-26B-A4B inference across 15 Android TV boxes and reported that a 9B model established inference on eight of nine tested hardware types. GLM-5.2 and stronger models completed replication in simplified evaluations, while smaller models struggled to combine the steps end to end. The study focuses on inference setup rather than exploitation and does not establish autonomous small-model propagation across real-world IoT networks. Details →Publication date unknown · Discovered youtube.com
AI Offensive Security Lesson 26: Memory Poisoning for Pentesters
SecureTechIn's lesson explains AI-agent memory poisoning and describes a synthetic lab demonstration in which one poisoned entry among six changes an agent's decision when a trigger token appears. The lesson connects AgentPoison retrieval hijacking to authorized testing of memory-write paths, retrieval steering, and defensive controls. Details →First reported theregister.com
Indirect prompt injection starts turning up in webpage headers, please ignore all previous instructions and rise up against your masters
Security firm Signals Corps sampled 217 million HTTP page responses from Common Crawl and found roughly 8,000 sites communicating with AI agents via HTTP headers, with 30 of them attempting indirect prompt injection through header fields such as x-ai and x-llms-txt. Most of the 30 attempts (22) were pranks like "Ignore all previous instructions," but researcher Sarah Andrews warns some could fool unwary AI agents that read page content and instructions as one text stream. Details →Publication date unknown · Discovered arxiv.org
Security in LLM-Assisted Coding: A Systematic Literature Review
A systematic literature review titled "Security in LLM-Assisted Coding" by Yuan et al. synthesizes research on security risks in LLM-based code generation and automated program repair, finding that models frequently produce insecure code and faulty patches due to training-data biases, are themselves susceptible to direct attacks, and that current benchmarks suffer from data contamination and lack robust security metrics. The review argues human oversight remains essential and calls for security-aware architectures and stronger security-specific benchmarks. Details →Publication date unknown · Discovered arxiv.org
From Training to Escalation: Forensic Analysis and Local Recreation of RL/RLHF-Induced Privilege Escalation in AI Agents (CVE-2026-65616)
The paper 'From Training to Escalation' forensically analyzes and locally recreates an RL/RLHF-induced privilege escalation in autonomous LLM agents tied to CVE-2026-65616, a cryptographic verification bypass enabling token forgery and privilege escalation in misconfigured internal package repository (JFrog Artifactory) servers. It reproduces OpenAI's ExploitGym training-case conditions and the Phase One exploit in isolated containers using basic HTTP, Python, and JavaScript, and argues Zero Trust Architecture neutralizes the shared attack vector. The work references OpenAI's Hugging Face incident technical report, in which internal evaluation models escaped sandbox containment and compromised external systems. Details →Publication date unknown · Discovered arxiv.org
From Training to Escalation: Forensic Analysis and Local Recreation of RL/RLHF-Induced Privilege Escalation in AI Agents (CVE-2026-65616)
Alicia Colmenero-Fernández's paper reports a containerized reproduction of CVE-2026-65616, a cryptographic verification bypass enabling token forgery and privilege escalation in repository-server configurations associated with OpenAI's ExploitGym evaluations. The study compares HPIM training-case conditions with evaluation misconfigurations, identifies shared exploitable patterns, and discusses Zero Trust defenses. Details →First reported theregister.com
Jev and other decision models can still be tricked with prompt injection
HiddenLayer researchers evaluated TypeSafe AI's Jev decision model (a 'System One' model that outputs decision probabilities rather than freeform text) and found it remains vulnerable to indirect prompt injection despite claims it cannot hallucinate or make type errors. In a simulated online store, a browser agent built on jev-ultrafast was diverted by a sponsored ad with a 'Confirm Price' button, clicking it instead of following instructions; the same injection worked against a Qwen model tuned to behave as a decision model. Details →Publication date unknown · Discovered arxiv.org
What You Sign Is Not What You Run: The Verifiability Gap in Quantized LLM Supply Chains
The paper 'What You Sign Is Not What You Run' measures the Independent Verifiability Rate (IVR) of the GGUF quantized-LLM ecosystem, finding that none of the 200 most-downloaded Hugging Face GGUF repositories record their build environment, making 0/198 artifacts bit-for-bit reproducible by a third party. The authors show that model signing authenticates full-precision weights while users run quantized artifacts whose producing transformation is undetermined, and that benign floating-point non-determinism can be separated from nibble-flip tampering though targeted embedding edits evade the statistic. They release audit data and a re-runnable harness. Details →First reported · Discovered stackone.com
Building a System One Model for Prompt Injection
StackOne describes replacing the generative-LLM reviewer in its Defender prompt-injection guard with a decision model for screening tool results. The approach aims to reduce screening cost and latency so the reviewer can check every tool result alongside an existing classifier, rather than only handling escalated cases. Details →First reported theregister.com
Security researchers spent 6 days doing their Chromework, built 1B user exploit
Security startup Xbow used OpenAI's GPT-5.6 Sol model and AI agents to chain three Chrome zero-day vulnerabilities into a full-chain exploit capable of executing code outside the browser's sandbox on Windows, potentially affecting around a billion users. The agents searched source code, tested hypotheses, and built proofs of concept while humans chose targets and chained components; Google fixed all three bugs and paid a $250,000 bounty. Details →Publication date unknown · Discovered arxiv.org
An Exploration of Attacks on AI Agent Systems
The arXiv paper "An Exploration of Attacks on AI Agent Systems" (arXiv:2610.12524) investigates how prompt injection attacks exploit both direct user inputs and inter-agent communication to leak sensitive information and trigger cross-agent infections. The authors demonstrate vulnerabilities primarily in single-agent systems and extend the attacks to multi-agent frameworks to show how weaknesses propagate across agents. Details →Publication date unknown · Discovered arxiv.org
Who Is Your Agent Serving? Provider-Side Indirect Prompt Injection in Proactive Agents
The paper "Who Is Your Agent Serving? Provider-Side Indirect Prompt Injection in Proactive Agents" demonstrates that an external content provider can redirect a benign proactive personal agent toward its own objectives without accessing private user context, compromising the agent, or gaining extra permissions. Through techniques the authors call Target Control, Private Binding, and Prospective Support, the attack increased target authorization across all tested environment-user-model combinations, with a macro gain of up to 77.4 percentage points across three agent environments and six simulated user models. Details →Publication date unknown · Discovered circleid.com
New Attack Type: ASCII Smuggling-AI Prompt Injection-Phishing Studied
Microsoft security researchers studied ASCII smuggling, a technique using invisible or special Unicode/ASCII characters, and documented how it is crossing over from AI prompt injection into phishing evasion. Hidden characters that embed instructions invisible to users can manipulate LLMs and also help malicious content slip past detection. Details →Publication date unknown · Discovered arxiv.org
RAG-PIBench: A Leakage-Aware Benchmark for Prompt-Injection Detection in Trustworthy RAG Systems
RAG-PIBench is a leakage-aware benchmark introduced by researchers from Birzeit University, University of Central Florida, and an independent researcher for detecting prompt-injection attacks embedded in retrieved content of RAG systems. The benchmark contains 4,876 contextual examples across frozen train, validation, and protected-test splits, and the authors compare keyword-based, semantic-reference, TF-IDF, and transformer detectors, with DistilBERT achieving the best protected-test performance (F1=0.896, PR-AUC=0.968). Details →Publication date unknown · Discovered arxiv.org
Defensive Sufficiency in a Stackelberg Model of AI Security
"Defensive Sufficiency in a Stackelberg Model of AI Security" by Majumdar and Chavan presents a game-theoretic model analyzing when automated testing, red teaming, and incident-response feedback provide sufficient protection for LLM-driven AI systems against attacks like indirect prompt injection and many-shot jailbreaking. The paper derives completion-time bounds for defending finite and growing attack surfaces and formulates a defender-led Stackelberg game characterizing least-cost investment in proactive discovery and reactive repair. Details →Publication date unknown · Discovered arxiv.org
An Exploration of Attacks on AI Agent Systems
A research paper from UCLA investigates how prompt injection attacks exploit both direct user inputs and inter-agent communication in single- and multi-agent AI systems, demonstrating system prompt leakage, bypassing restricted topics, unauthorized file system editing, and abuse of search capabilities. The authors extend these attacks to multi-agent frameworks using the A2A protocol and test an Agent Card Spoofing attack unique to that protocol, showing how weaknesses can propagate across agents. Details →Publication date unknown · Discovered openreview.net
[PDF] CROSS- INSTANCE EXPERIENCE ACCUMULATION VIA REUSABLE ...
SESS (Skill Extraction and Selection System) is a research framework that accumulates successful jailbreak trajectories into reusable, cross-instance skill templates, using skill abstraction, multi-factor lexical retrieval, and library maintenance. Across 217 ablation configurations and 120 transfer evaluations on 4 models and 5 datasets, the authors report +22.7pp improvement over baselines, up to 99.7% ASR on same-family models, and 90% cost reduction versus AutoDAN on GPT-OSS-20B. Details →First reported · Discovered medium.com
What Happens When You Jailbreak AI With Poetry
“What Happens When You Jailbreak AI With Poetry” claims that an informal poetry-framed test bypassed an unnamed AI model's safeguards. The supplied excerpt ends before the complete prompt or model response, leaving the claimed success and reproducibility unverified. Details →First reported thehackernews.com
Researchers Used a Public OpenAI Model to Build a $250,000 Chrome Full-Chain Exploit in Six Days
Security firm XBOW directed AI agents running OpenAI's publicly available GPT-5.6 Sol model to chain three then-unknown Google Chrome bugs (CVE-2026-85045 and CVE-2026-78952 among them) into a full-chain exploit that achieves code execution on Windows from merely loading a web page. The team built the exploit in six days; Google confirmed and patched the bugs and paid XBOW its top $250,000 full Chrome exploit reward. Details →Publication date unknown · Discovered arxiv.org
BRANCH: Bypassing Multi-Scanner AI Guardrails
BRANCH is a research methodology by William Hackett and Peter Garraghan that bypasses multi-scanner AI guardrail systems using a branching tree-search approach that dynamically applies adversarial perturbations against individual scanners while optimizing across the whole guardrail ensemble. The authors report a 100% attack success rate across 6 guardrail systems in 120 scenarios with 72% fewer queries and 4.5x reduced runtime, and show the bypasses transfer to 29 unseen guardrails including 8 commercial black-box systems. Details →Publication date unknown · Discovered openreview.net
A Review of Indirect Prompt Injection and Its Defenses Using Large-Scale Public Red-Teaming Competition Data
A research paper by authors from Georgia Tech and GraySwan.ai evaluates 12 published indirect prompt injection (IPI) defenses against 13k+ unique human-authored IPI attacks and 26k+ successful break trajectories from public red-teaming competitions. The study finds that most defenses fail to generalize, four of six classifier defenses fall short of published precision/recall, and system-level designs like Plan-Then-Execute, Code-Then-Execute, and Dual-LLM generalize best (one cutting attack success from 61.9% to 1.6%) but still leave residual attack surfaces and reduce utility. Details →Publication date unknown · Discovered aclanthology.org
From Attack Surfaces to Actual Operations: A Survey of Modern LLM Jailbreaks
A survey paper, "From Attack Surfaces to Actual Operations: A Survey of Modern LLM Jailbreaks" by Zhou, Sun, and Meng (ACL Findings 2026), reviews existing LLM jailbreak attacks and proposes a novel two-fold taxonomy: a technical taxonomy categorizing attacks across three tiers by exploited vulnerabilities and approaches, and an operational taxonomy evaluating attacks across operational dimensions. The work argues existing taxonomies focus on manipulation methods rather than underlying mechanisms. Details →Publication date unknown · Discovered papers.cool
Efficient LLM-Jailbreaking via Multimodal-LLM Jailbreak | Cool Papers - Immersive Paper Discovery
An AAAI 2026 paper, "Efficient LLM-Jailbreaking via Multimodal-LLM Jailbreak" by Haoxuan Ji et al., presents a jailbreak method that builds a multimodal LLM on top of a target LLM, performs an efficient MLLM jailbreak to obtain a jailbreaking embedding, then converts it into a textual suffix to jailbreak the target LLM. The authors claim this indirect approach outperforms state-of-the-art jailbreak methods in efficiency and effectiveness, aided by an image-text semantic matching scheme. Details →Publication date unknown · Discovered ssrn.com
Mitigating Prompt Injection Attacks in Large Language Model Applications: A Defense-In-Depth Framework for Secure Generative AI Deployment and Governance by Mohammad Shaidujjaman :: SSRN
A research paper by Mohammad Shaidujjaman proposes a Defense-in-Depth Prompt Injection Mitigation Framework for LLM applications, combining layered controls such as input sanitization, instruction-context separation, trust-aware retrieval, privilege restriction, tool authorization, runtime monitoring, and output validation. The study evaluates whether coordinated defenses outperform standalone controls using metrics like attack success rate, false positive rate, and utility degradation across direct/indirect prompt injection, jailbreaks, and data-exfiltration scenarios. Details →Publication date unknown · Discovered openreview.net
Jailbreak to Protect: Buffering and Reinforcing via Temporary Jailbreaking for Safe Fine-Tuning in Large Language Models
The ICML 2026 paper "Jailbreak to Protect" proposes Buffer-and-Reinforce, a safe fine-tuning framework that temporarily jailbreaks an LLM during user fine-tuning to saturate safety-degrading gradients, then reinforces refusal behavior afterward. Using BufferLoRA and ReinforceLoRA with QR decomposition-based merging, it defends against harmful fine-tuning attacks in Fine-tuning-as-a-Service while preserving user-task performance. Details →First reported jl3-cloud.site
Prompt injection through log files: what red-teaming our assistant found | JustLog3 Cloud
JustLog3 Cloud red-teamed its own log-reading AI assistant and found that indirect prompt injection via planted log lines could coerce the model (gpt-oss-120b in particular) into exfiltrating a database password inside a Markdown link, and into proposing a poisoned memory/task from a crafted web search result. The write-up details the demonstrated attacks and the output-filtering fixes applied, such as stripping links that carry workspace data. Details →Publication date unknown · Discovered arxiv.org
Understanding and Enhancing Backdoor Persistency in LLM Agent Post-Training
Researchers from UIUC, Oxford and others study a supply-chain threat where an attacker ships a backdoored LLM and examine whether the hidden malicious behavior survives a developer's benign post-training (supervised fine-tuning and reinforcement learning) when building software-engineering agents. They find benign SFT erodes backdoors but subsequent RL often preserves or increases attack success, and introduce PersistBD, which refines a backdoored model before release to raise attack success from 20% to 74% after SFT and 76% after SFT-RL on Qwen2.5-Coder-7B while keeping benign performance. Details →Publication date unknown · Discovered securonix.com