Research
The Provenance Tax: Understanding the Impact of LLM Watermarking on AI Agent Behavior
Publication date not yet evaluated · Added lasso.security
Page published
Coverage timeline
Why it matters
Watermarking mandated for provenance can inadvertently weaken safety refusals and change agent tool-calling behavior, meaning a compliance-driven change could make deployed LLM agents more susceptible to prompt injection and unsafe actions.
Lasso Security researchers studied how LLM text watermarking—specifically Google DeepMind's SynthID-Text now being deployed by Anthropic's Claude to comply with the EU AI Act—alters model behavior at generation time, an effect they call 'sampling drift.' Because watermarking changes how each next token is sampled, they empirically find it can shift both model refusal behavior (including whether refusals hold under prompt injection) and agent tool-calling decisions (which tools get called and with what arguments), with effects that are model- and key-dependent.