AWS security researchers published a benchmark on September 9 measuring something most AI-powered vulnerability scanners avoid reporting on themselves: how often they flag safe code as dangerous. The Deception Benchmark ran 12 models from five providers, including GPT-5-series, Claude, Llama 3.3, Amazon Nova, and Mistral Large, against 14,822 code samples across 16 languages and more than 70 vulnerability categories, and none of them met AWS’s own production bar of under 10 percent false positives and under 10 percent false negatives.
When prompted directly to flag vulnerable code, models caught most real flaws but misfired on 41 to 99 percent of safe code, with precision as low as 52 percent. Pushing models to prove a flaw was actually exploitable, rather than pattern-matching it, cut false positives by 17 to 74 percentage points, but missed 7 to 44 percent of genuine vulnerabilities in return. The best performer, Claude Opus 5, still carried a 41.5 percent false-positive rate against a 5.2 percent false-negative rate.
Why it matters for the security leader: false positives erode trust in AI-assisted triage until analysts start ignoring the tool altogether, the same alert-fatigue failure that has plagued static analysis for years. As AWS researchers Anshumali Shrivastava and Neha Rungta wrote, “Production tools rely on multi-step loops and agentic workflows to compensate, but that scaffolding masks whether the model itself understands the code.” The original insight worth sitting with: the industry has been measuring AI vulnerability tools by how many attempts an agentic harness takes to reach a verdict, not whether the underlying model actually understands the code, a different and harder question than AI-AppSec marketing generally admits.
CyberTech has tracked this gap before, including the mismatch between predicted and observed AI malware volume and Anthropic’s documented case of AI-run attack operations. Security leaders piloting AI-assisted triage should demand a false-positive and false-negative rate from any vendor, not just a detection-rate claim, and treat single-pass model judgment as a signal for a human reviewer, not an autonomous gate.
Source: AWS Security Blog
