The bug every AI coding tool ships, and how to prove it is gone

The scanner is open source and the rules are readable before you trust a single finding: github.com/audit0/auditai-scanner. This post is about the one bug it was built for. The shape of the bug Ask any AI coding tool for an invoicing app and you will get something close to this route handler. // app/api/invoices/[id]/route.ts import { createClient } from "@supabase/supabase-js";const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_SERVICE_ROLE_KEY!); export async function GET(_req: Request, { params }: { params: { id: string } }) { const { data } = await supabase.from("invoices").select("*").eq("id", params.id).single(); return Response.json(data); } It works. It passes review if the reviewer is reading for "does this return an invoice". The service-role key bypasses row level security, and the query filters on id alone, so Alice can read Bob's invoice by changing a number in the URL. This is not an exotic bug. It is the default outcome of asking for a feature and not asking who is allowed to see it, and it shows up over and over in apps built with Lovable, Bolt, v0, Cursor and Claude Code. Why a warning is not enough A scanner that prints "possible authorization issue" at this line has told you almost nothing. You still have to open the file, work out whether the route is reachable, whether some middleware already blocks it, and whether the id is scoped somewhere you did not read. Most teams do that work once, find two false positives, and stop reading the tool's output. "The model says it fixed it" is worse. A model that both writes the fix and grades the fix is not evidence of anything. So we picked a harder bar. A finding stays likely until something outside the model reproduces it. What reproduction looks like We copy the app into a sandbox with no internet access, seed two synthetic tenants, and send the same request twice. proof · AUDIT-001 GET /api/invoices/42 as Alice ✗ before fix: 200 OK ✓ after fix: 403 Forbidden Alice is not supposed to see invoice 42. Before the fix the endpoint hands it over. That is the evidence. It is also, conveniently, a test. The fix and the regression test The fix is the smallest one that closes the path, not a refactor: -const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_SERVICE_ROLE_KEY!); +import { createServerClient } from "@/lib/supabase/server"; export async function GET(_req: Request, { params }: { params: { id: string } }) { - const { data } = await supabase.from("invoices").select("*").eq("id", params.id).single(); + const supabase = await createServerClient(); + const { data, error } = await supabase.from("invoices").select("*").eq("id", params.id).single(); + if (error || !data) return new Response("Forbidden", { status: 403 }); return Response.json(data); } The request now runs as the signed-in user, so the table's row level security policy decides the answer instead of the route. The regression test is the reproduction, kept: it("does not let one tenant read another tenant's invoice", async () => { const res = await fetch(`/api/invoices/${bobInvoiceId}`, { headers: aliceAuth }); expect(res.status).toBe(403); }); It failed before the fix. It passes after. If someone reintroduces the service-role client next quarter, it fails again. The honest part Every result ends with a coverage line rather than a score: checked 12 routes · verified 2 · confirmed 1 · unverified 0 unverified is not a failure to hide. It is the set of findings we could not reproduce, and we would rather show you the number than dress it up. Findings move from candidate to likely to confirmed only on evidence, and a fix is verified only when the regression test failed before it and passes after it, the existing suite still passes, and a deterministic rescan no longer sees the path. We measure precision on repositories we have never seen, and publish the number instead of claiming one. Try it npx auditai-scan . It runs locally, needs no account, and sends nothing anywhere. Source, rules and the eval fixtures are here: github.com/audit0/auditai-scanner. The hosted product that reproduces, fixes and proves is at auditai.sh. If you have a Next.js + Supabase app and want us to find and prove one real authorization bug in it for free, ten design-partner slots are open.