I published our agent-security benchmark, including the attacks we fail to catch

Every company building AI agent security publishes a detection rate. We were doing it. The problem is that none of these numbers are checkable. You see "99% detection" or "blocks 100% of prompt injection," and there is no test set, no failure list, no methodology, nothing to run. You are asked to trust it. I don't think trust is the right thing to ask for in security. So I published our benchmark, with the parts that don't flatter us left in. It's here: https://github.com/AndrewSispoidis/contemporary-agent-attacks Here is what is in it. 497 attacks and 1,172 benign samples, 1,669 total, across 22 categories. The detection engine scores 99.8% on the attacks at a 0.09% false positive rate. The license is CC BY 4.0. The methodology and scoring code are in the repo. You can clone it and reproduce every number. The part I care about more is the failures. The benchmark names them. There is one attack the engine misses, a pretext opener that never actually asks for anything it shouldn't, so there is nothing to catch at the point it is measured. There is one benign sample it flags, a Stack Overflow question about Go method-receiver syntax that has the shape of a probe without being one. Both are in the repo by name. If you run it, you will hit exactly those two and no surprises. Here is why I think this matters for the category as a whole. When detection numbers can't be reproduced, they stop carrying information. A vendor can claim anything, and the buyer has no way to check, so the decision comes down to who markets hardest. That is a bad way to choose a security control. A benchmark you can run, that is honest about where it breaks, gives you something real to evaluate. I would rather compete on numbers people can verify than on numbers people have to believe. If you work on agent security, or you are trying to evaluate it, clone the benchmark and tell me where it is wrong. The failures are where the useful conversation is.