Research
Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation
Publication date unknown · Discovered arxiv.org
Page published
Publication date unknown · First observed: 30 Oct 2026
Coverage timeline
Single-source research — one report is available.
Why it matters
The semantic reconstruction study shows how evaluator choice can distort jailbreak results, helping defenders assess whether apparent safety failures reflect substantive compliance rather than refusals or merely related content.
“Did the LLM Answer the Question?” reports that changing only the evaluator LLM can shift measured jailbreak attack success rates by up to 82 percentage points. The paper introduces a training-free semantic reconstruction game that reconstructs the request answered by a response and compares it with the original and similar decoy requests. Across three LLM judges, the method achieved the lowest mean error against human labels among evaluated methods, prioritizing fewer false positives at the cost of missing some successful jailbreaks.