My Scanner Missed 93% of the Bugs — and That Was the Right First Result

The first time I ran my vulnerability scanner against the industry-standard benchmark, the bottom line of the scorer's report was this: $ python scripts/score_benchmark.py --findings out/java.findings.json \ --truth benchmark-java/expectedresults-1.2.csv OVERALL precision 0.60 recall 0.07 F1 0.13 # abridged Three numbers, and here is what each one means. Precision 0.60 — of all the alarms the scanner raised, 60% pointed at real bugs: when it spoke, it was right more often than not. Recall 0.07 — of all the real bugs in the benchmark, it found 7%. In the four vulnerability classes my scanner covers, the benchmark contains 777 real, labeled vulnerabilities; it missed 93% of the bugs it exists to find. F1 0.13 — precision and recall combined into one score (their harmonic mean), dragged down to almost nothing by that recall. My first instinct was to fix it before anyone saw it. Instead I saved the output, wrote the number into my benchmark log, and kept it — because that number was always going to be published, and this is the article that publishes it. The Context For the past months I've been deep in AI — reading, building, measuring. One of the projects that came out of it is an AI vulnerability scanner. The design in one sentence: deterministic static-analysis rules do all the searching, and an LLM judges each finding — is this a real bug or a false alarm? The full architecture gets its own article. This one is about the first measured number. The test set is the OWASP Benchmark — 2,740 labeled Java test cases, the standard exam for Java security scanners. In my scanner's four vulnerability classes (SQL injection, command injection, path traversal, XSS) there are 1,478 cases: 777 real vulnerabilities and 701 cases deliberately designed to bait scanners into raising false alarms. Every tool I compare against — Semgrep, CodeQL — takes the same exam, scored by the same scoring code. Same rules for everyone. New to this? Three words carry this article. A source is where untrusted input enters a program (request.getParameter("id") — anything an attacker can type). A sink is where that input becomes dangerous (executeUpdate(sql) — running it as a database command). A vulnerability is data flowing from a source to a sink without being cleaned on the way; that flowing data is called tainted, and tracking it is taint analysis — the scanner's whole job is finding those flows. Already know all that? Skip this box. Why 0.07 Was the Right First Result Here's what I had actually run: a spike. A deliberately minimal first version — one source pattern, getParameter, wired to a handful of sinks, pushed end-to-end through the whole pipeline: parse 2,740 test cases into a code graph, run the taint queries, emit findings, score them against the answer key. The spike's job was never to score well. Its job was to answer one simple question: does the machinery work at all? And the ugly number, read carefully, answered it: Precision 0.60 — when the scanner did raise an alarm, it was usually right. The taint engine was tracking real flows correctly. Recall 0.07 — it was blind to 93% of the bugs. The engine wasn't broken; its vocabulary was tiny. I was listening at one door of a building with many doors. That's not a broken idea. That's a correct, simple diagnosis of a too-narrow source list — delivered before I had invested weeks in the wrong layer. If the first number had been precision 0.10, I'd have had an engine problem, which is a rebuild. A recall problem is a list problem. Lists are fixable. Why the Obvious Moves Are Both Wrong Obvious move #1: don't tell anyone. Fix it quietly, publish only the final number, look competent. Almost everyone building in public does a version of this — the "overnight" success graph that starts at the first good result. The problem: every result in this series is a number from my own benchmark runs. There is no referee here — no third party checks my work before it ships; it is me, a scorer script, and you. A reader has exactly one way to judge numbers like that: the author's track record with results that hurt him. If every number I show you is a win, you have no reason to trust any of them. So the bad numbers ship too — and they ship first. I'm going to publish a head-to-head against Semgrep and CodeQL later, and when I claim a result there, I want the reader thinking "this is the person who published his own 0.07." Honesty is not a virtue here; it's infrastructure. Obvious move #2: carpet-bomb the rules. Recall too low? Add patterns! Match more names, loosen the regexes, taint everything — recall will climb. It will also destroy precision, and worse: after twenty simultaneous changes you cannot say which one did what. You've traded a measured system for a vibes system. What I did instead was slower and duller: read the benchmark's actual code, find what it really calls, add sources in order of how often the code uses them, and re-measure after every change. One variable at a time, one number per change. The Climb Fix 1: Learn the benchmark's vocabulary — recall 0.07 → 0.83 I surveyed which input methods the benchmark's code actually uses, counting files: getRequestURI in 724 files, getCookies in 664, getParameter in 538, getParameterValues in 510, getHeaders in 400, and on down the HTTP request surface. My spike had covered exactly one entry in that list. So sources became one shared definition — a single regex over fully-qualified method names, so each getter is bound to the type that makes it attacker-controlled. Condensed here; the full rule table gets its own article: // Queries run on Joern, an open-source code-analysis engine (Scala DSL). ".*(HttpServletRequest|ServletRequest)\\.(getParameter|getParameterNames|getHeader|" + "getHeaders|getCookies|getQueryString|getRequestURI|getInputStream|…)\\b.*" + "|.*Cookie\\.(getValue|getName)\\b.*" + "|.*SeparateClassRequest\\.(getTheParameter|getTheValue)\\b.*" // the benchmark's request wrapper (.*Cookie\.getValue.* matches only the cookie's getter — a bare match on getValue would match every getValue in existence.) Two more problems were hiding inside this step, and they were different problems: 117 of the benchmark's real SQL injection cases never touch java.sql — they go through Spring's JdbcTemplate instead. One new sink row took SQLi recall from 0.57 to 0.86. 6,060 XSS sink calls were invisible for a reason outside my rules. Without the servlet library on the analysis path, the engine cannot work out what type response.getWriter() returns, so those calls could never match a type-based .*Writer.* pattern. The fix: also accept a sink when the receiver text — the response.getWriter() part as literally written in the code — matches getWriter|getOutputStream. That single change took XSS recall from 0.03 to 0.73. New score: precision 0.53, recall 0.83. Fix 2: 97 of the 130 remaining misses shared one missing source — 0.83 → 0.95 There were still 130 real bugs missing. I diffed the misses against the benchmark code, and 97 of them — three quarters of everything left — took their taint from a single method I hadn't listed: getParameterNames(). It's easy to see why it gets skipped. getParameter("id") returns a value the user typed — obviously dangerous. getParameterNames() returns the parameter names — and names feel like structure, not data. But the client chooses the names too. ?alert(1)=x is a query string any client can send, and then the name is exactly as attacker-controlled as the value. One name in a regex. Adding it recovered 93 real vulnerabilities on the spot: recall 0.83 → 0.95. (The other 4 of the 97 were also blocked by a second, separate problem — they return in Fix 3.) And here's the part that still bothers me: nothing ever crashed, warned, or looked wrong. A missing source fails silently. Without a labeled benchmark I would never have known. Fix 3: The taint bridge — 0.95 → 1.00 That left 37 misses — the 33 that never used getParameterNames, plus the 4 from Fix 2 that had a second problem — and every single one contained .split(...). Instead of guessing, I measured the taint chain link by link on one failing case: the variable being split was reachable from the source. The split call itself — reachable. The array-index access on its result, param.split(" ")[0] — not reachable. Taint flowed correctly through split and died at the index operation. The engine ships a default rule for that operator, and overriding it changed nothing — the gap was in how the engine applies rules to that operator, not in anything I could configure away. So I built a bridge, with one condition that keeps it honest: an index access over a split-style call is promoted to an additional source only when the array it indexes is itself reachable from a real source. Indexing an untainted array stays untainted — that condition is the difference between a targeted fix and blanket over-tainting. All 37 misses recovered, at a cost of exactly three new false positives and about 84 seconds of extra work per scan — a fix that raised recall and precision at the same time. The Numbers The whole climb, one measured change at a time: stage precision recall F1 spike (getParameter only) 0.60 0.07 0.13 + broadened sources, receiver-text sinks 0.53 0.83 0.65 + getParameterNames 0.55 0.95 0.70 + index-access taint bridge 0.56 1.00 0.72 Zero false negatives. All four classes land at recall 1.00 — SQL injection 272 of 272 real bugs found, command injection 126 of 126, path traversal 133 of 133, XSS 246 of 246 — the same recall CodeQL achieves on the same 1,478 cases, scored by the same code. From a seven-row rule table, with no build step. Now the uncomfortable part, because this is an engineering log and not a launch post. Precision 0.56 means 614 false alarms, and this layer falls into 88% of the benchmark's designed traps — more than either incumbent (the head-to-head article prints the full comparison). Perfect recall with weak discrimination is uncomfortably close to a tool that says "maybe" about everything, and I'd rather write that sentence myself than have a reader write it for me. Three objections, answered before you raise them "You fixed recall by reading the benchmark's own code and adding whatever it calls. That's tuning on the test set — recall 1.00 means nothing." Partly right, and worth being precise about. The benchmark is public, and it is open-book for every tool measured on it — the incumbents tune against it too. But look at what the fixes actually were: the standard HttpServletRequest input surface — getHeader, getCookies, getRequestURI — not benchmark-specific hacks; the one special case, the benchmark's own wrapper class, is disclosed in the regex above. And the traps argue against gaming: I fall into 88% of the designed false-alarm cases, worse than CodeQL — if I were fitting to the answer key, that is the first number I would have fixed. What recall 1.00 honestly claims is narrower: given a vocabulary, the engine misses nothing that speaks it — and the loop that built the vocabulary (survey what the code actually calls, add sources by count, re-measure after every change) is the same loop that would onboard any real codebase. Generalisation to real repositories is unmeasured; when I measure it, it gets published, good or bad. "Why not just use CodeQL? Same recall, better precision." True, and the head-to-head article prints that row in plain text. Two reasons this project exists anyway. First, this discovery layer is a seven-row rule table with no build step — CodeQL brings years of modeled libraries and wants your build. Second, the precision problem is deliberate surplus: discovery over-reports so that the judge — the LLM layer — has material to remove, and in controlled tests, the best judge model I tried removed half the false alarms — 52% in the latest verified run — at a cost of 2% of the real bugs. The comparison that decides the design is judged output against the incumbents, and that number gets published either way. "An LLM in the pipeline means the results can't be reproducible." The non-determinism is quarantined by design. Everything in this article is the deterministic layer: same code in, same flows out — the whole 0.07 → 1.00 climb involved no LLM at all. The model never searches; it only rules on findings the rules already produced. And its verdicts are not trusted — they are measured against labeled ground truth, on a fixed-seed sample, with confidence intervals. How that measurement went — and why the choice of judge model mattered far more than I expected — is a later article. Going deeper (skip freely — statistics only) Recall 1.00 against a false-positive rate of 0.88 gives this layer a Youden's J — recall minus false-positive rate, a one-number score that punishes saying "maybe" to everything — of just 0.12. Weak discrimination, stated plainly; raising that number is precisely the judge's job. And the judge figures quoted above are estimates from a 200-candidate stratified sample, which is why they carry confidence intervals — sample-level and full-census numbers never share a table in my logs, because mixing the two silently shifts precision by a point. What I Learned 1. Ship the embarrassing number. Publishing 0.07 cost me nothing — the scanner improved just as fast either way. What it bought is the right to be believed later: when the head-to-head against Semgrep and CodeQL comes out, every number in it is backed by the fact that I publish the ones that hurt. 2. A spike's job is to be wrong, cheaply. The minimal version told me in one run which layer was weak (vocabulary, not engine) — before I had built anything expensive on top of it. If your first measurement of a new system isn't a little embarrassing, you probably measured too late. 3. Coverage failures are silent — only measurement finds them. 97 of my 130 misses traced back to one absent source, without a single crash, warning, or odd log line. This goes far beyond security: the error path nobody tests, the market segment nobody surveys, the input case nobody generates — absence never announces itself. If you aren't measuring against ground truth, your blind spots don't feel like blind spots. What can you do with this? The lessons above are mine; this section is yours. Take it by who you are: You build software — any software. Two habits you can apply this week: check whether your input validation covers the names of inputs, not only their values (the client chooses both — that gap hid 97 bugs from me); and for any system you own, ask "where is my labeled answer key?" A missing capability produces no error and no log line. Only measurement against known truth makes blind spots visible. You build with LLMs or any detection system. Steal the loop wholesale: ship a minimal spike end-to-end, measure it against ground truth, change one variable, re-measure, keep every number. It located my weakest layer in a single run, and it will locate yours. You're learning security. You now hold the four ideas that organize static analysis — source, sink, tainted flow, and the precision/recall trade. Everything here is reproducible: the deterministic layer runs completely free, with no API key needed. That's the first number. Coming in this series: the architecture the whole scanner stands on — why the rules do all the searching and the AI only judges — the judge experiments, and the full head-to-head against Semgrep and CodeQL. Every number gets published, especially the bad ones. (Code and benchmark artifacts go public alongside the head-to-head article.) I'm Ali Afana — AI builder and security researcher, writing from Gaza. I build systems in public, measure them against ground truth, and keep the receipts. This scanner is one project on a longer road — follow for what comes next. GitHub · X · LinkedIn