Research
How Your Credentials Are Leaked by LLM Agent Skills: An Empirical Study
Publication date not yet evaluated · Added · 6 reports arxiv.org
Page published · Page updated
Coverage timeline
Why it matters
LLM agents with access to emails, databases, and third-party skills can leak credentials and sensitive data even during benign, non-adversarial use, making operational data leakage a first-order agent-security concern for enterprise defenders.
Two research studies examine how tool-using LLM agents leak sensitive data. A joint Singapore/Korea AI Safety Institute evaluation found that across 12 realistic non-adversarial tasks, none of three tested agents executed all scenarios safely, frequently disclosing information to inappropriate recipients or accessing unnecessary data. A separate empirical study of 17,022 sampled skills from the SkillsMP marketplace identified 520 affected skills with 1,708 security issues, finding that debug logging into the LLM context window accounts for 73.5% of credential exposures and that 89.6% of leaked credentials are immediately exploitable.
Summary
Researchers present what they describe as the first large-scale empirical study of how credentials are leaked by third-party LLM agent skills. Drawing on 170,226 artifacts from SkillsMP, the largest open-source skill marketplace, they analyzed a stratified random sample of 17,022 skills using static secret extraction (regex and AST parsing), dynamic sandbox testing with mock credentials, and cross-referencing of developer intent against runtime behavior.[1]
The analysis found 520 affected skills containing 1,708 security issues and derived a taxonomy of 10 leakage patterns. Key findings include that credential exposure is fundamentally cross-modal (76.3% of cases require analyzing both natural-language descriptions and code), that debug logging is the dominant vector (73.5% of vulnerabilities) because agent frameworks feed stdout into the LLM context window, and that 89.6% of leaked credentials are immediately exploitable, with 92.5% exposed during routine execution without elevated privileges.[1]
The work is research rather than observed in-the-wild exploitation. The authors conducted responsible disclosure, after which all malicious skills were removed and 91.6% of hardcoded credential cases were remediated, and they released their dataset, taxonomy, and detection pipeline. The paper was accepted to ASE 2026.[1]
Disclosure timeline
| Date | Event |
|---|---|
| 2026-04-03 | Initial version (v1) of the study submitted to arXiv (arXiv:2604.03070).[1] |
| 2026-06-19 | Revised version (v2) submitted to arXiv; paper accepted to ASE 2026.[1] |
How it works
Agent skills operate within privileged execution environments and routinely handle sensitive credentials. The dominant leakage mechanism is debug logging: because agent frameworks feed process stdout into the LLM context window, routine debug output that prints secrets becomes a credential exposure vector, accounting for 73.5% of the vulnerabilities identified.[1]
Credential exposure in skills is cross-modal — 76.3% of cases require jointly analyzing natural-language descriptions and programming logic to detect, meaning code-only or description-only scanning misses most cases. The study catalogued 10 distinct leakage patterns across 520 affected skills (1,708 issues total).[1]
Remediation is undermined by the marketplace's fork-based distribution model: secrets removed from 107 upstream repositories persisted across more than 50 independent forks, so upstream cleanup does not eliminate exposed credentials downstream. 89.6% of leaked credentials were immediately exploitable and 92.5% were exposed during routine, unprivileged execution.[1]
Affected versions and patch status
| Product | Affected | Patch status |
|---|---|---|
| SkillsMP open-source agent skill marketplace | 520 skills identified as affected (1,708 security issues) out of a 17,022-skill sample drawn from 170,226 marketplace artifacts | After responsible disclosure, all malicious skills removed and 91.6% of hardcoded credential cases remediated; residual secrets persist across 50+ forks[1] |
Key takeaways
- Third-party LLM agent skills are a material credential-leakage surface: a sample-based study found 520 affected skills with 1,708 issues, most exploitable during ordinary use without elevated privileges.[1]
- Routine debugging is a security hazard in agent frameworks because stdout is piped into the model's context window, so credentials printed for debugging become directly exposed.[1]
- Fork-based marketplace distribution makes remediation incomplete — removing a secret upstream leaves it live across dozens of forks, so credential rotation, not just deletion, is required.[1]
Defensive actions
- Suppress or scrub credentials from skill debug/stdout output before it is fed into the LLM context window.: Debug logging into the context window was the single largest leakage vector, responsible for 73.5% of the vulnerabilities identified.[1]
- Scan agent skills using combined analysis of natural-language descriptions and code rather than code-only or description-only checks, and adopt the authors' released detection pipeline and taxonomy.: 76.3% of credential-exposure cases are cross-modal and only detectable by jointly analyzing descriptions and logic; the authors released their dataset, taxonomy, and detection pipeline to support this.[1]
- Treat any credential ever committed to a skill repository as compromised and rotate it, tracking forks as well as upstream repositories.: The fork-based distribution model defeats remediation — secrets removed from 107 upstream repositories persisted across 50+ independent forks, and 89.6% of leaked credentials were immediately exploitable.[1]