Tool
VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green
First reported arxiv.org
Page published
Earliest dated coverage: 15 Oct 2026 · First observed: 15 Oct 2026 · Latest dated coverage: 15 Oct 2026
Coverage timeline
Single-source research — one report is available.
Why it matters
VulnBump gives defenders a runnable benchmark to measure whether AI remediation agents actually remove vulnerabilities rather than just avoiding breakage, exposing that unaided agents often claim fixes a rescan still finds.
VulnBump, a benchmark published by Snyk (with code and data on GitHub and an arXiv paper), evaluates whether AI coding agents can upgrade vulnerable open source dependencies across six package managers while keeping builds and tests green. Across 307 real upgrade tasks covering 2,066 fixable vulnerabilities, a non-agentic baseline passed 46.9%, Claude Code unaided 59.6%, Claude Code with Snyk scan 77.5%, and Snyk's purpose-built Evo Remediation Agent 90.2%.