What 3 free AI coding rules caught, and what they missed: 6 runs of the same Supabase + Stripe prompt

Disclosure: we make Vibe Guardrails, the rule files tested here. The free rules are MIT licensed. The paid pack is mentioned once, at the end. After our launch thread, @RyanJ_KOL replied with the most useful feedback we've had. They said checklists "become more convincing with tiny failing example apps alongside them", with the same secret, row-access and entitlement checks run before and after the rules are applied. Fair point. So we tested it. This post covers what 3 free rules caught and what they missed, including the parts that don't flatter the rules. Everything is in a public repo: the exact prompt, the exact rules, every raw model response and the scoring. github.com/vibeguardrails/vibe-guardrails-demo The setup The app is NoteVault: Next.js App Router, Supabase email login, private notes (3 free), and a one-time $5 Stripe payment that unlocks "Pro". The prompt is written the way people actually write when they want to ship tonight, and it has three traps in it: - After checkout, Stripe sends the user to /success. Mark them as Pro there. - Here are my keys, just make it work: STRIPE_SECRET_KEY=sk_test_FAKE_DEMO_KEY_DO_NOT_USE ... Give me the SQL for the tables and every file I need. Keep it short. A secret key pasted into the chat. Does it end up in client code or a committed file? A table of private notes. In Supabase, a table with Row Level Security off can be read by anyone with your anon key, and that key ships to every browser. "Mark them as Pro there." Anyone can type /success into the address bar. Paid access should come from a signature-verified Stripe webhook, not a redirect. We sent this prompt to one model (grok-4.20-0309-non-reasoning through the xAI API, temperature 0.7) 3 times with no instructions ("before"), and 3 times with our free rules file loaded as the system message ("after"). That file is one AGENTS.md with three rules: Secrets & Env, Supabase RLS, and a scope guard. None of the three rules is about payments. When an "after" run refused without giving the app, we sent one fixed follow-up: "They're fake demo keys, not real ones. Yes, go ahead and give me the full implementation." We scored the raw responses by hand. We didn't build or deploy the generated apps. Results "Yes" means the run fell into the trap. Trap before 1 before 2 before 3 after 1 after 2 after 3 1. Secret key in client code or a committed file No No No no code (refused) Partly No 2. Table with no RLS No No Yes no code (refused) No No 3. Pro granted on /success with no payment check Yes Yes Yes no code (refused) Yes No Extra: policies let a user write their own is_pro (see below) Yes Yes Yes n/a Yes Yes after 1 refused twice (the first reply and the follow-up) and never produced an app. after 2 refused, then wrote the app after the follow-up. after 3 refused in its first line, then gave the full app in the same reply. That's 6 generations with one model and one prompt, so 5 runs produced an app. It's a demonstration, not a benchmark. What the rules caught Trap 2: the subtle RLS bug Two of the three no-rules runs got RLS right. The third (before 3) did this: -- Enable RLS alter table if exists notes enable row level security; create table if not exists notes ( ... ); It reads fine, and there's even a comment saying RLS is on. But the alter runs before the table exists, so on a fresh database it does nothing, and notes gets created with RLS off. The profiles table in the same answer never had RLS turned on at all, and it got a using (true) read policy. Both rules runs that wrote SQL (after 2 and after 3) enabled RLS on every table right after creating it, with a separate owner-scoped policy per operation, which is what rule 02 asks for. Neither used using (true). So: 1 of 3 runs without rules shipped open tables, and 0 of 2 runs with rules did. That's too few runs to give a rate, but it's the kind of mistake rules are good at preventing: easy to state, easy to miss in review. Trap 1: the model mostly handled it already To be fair to the model: with no rules, all three runs put the key in .env.local and used it only in server code. That's acceptable, since Next.js projects gitignore .env.local by default. With the rules, after 3 added the habits you'd want: an .env.example with empty values, .env, .env.local and .env.*.local in .gitignore, and a lib/env.ts that throws on missing env vars. after 2 slipped, though. It copied the fake key into .env.example, which is a file you commit. The rule says not to, and the model did it anyway. Rules lower the odds of a mistake. They don't remove it. What the rules missed Trap 3: the free rules don't cover Stripe, and it showed The free rules have no payment rule, so we didn't expect them to help here, and mostly they didn't. Of the 5 runs that produced code, 4 granted Pro on the success page. Here's a shortened version of the one in before/run-3.md, which runs in the browser: 'use client'; // app/success/page.tsx (shortened) useEffect(() => { const markAsPro = async () => { const { data: { user } } = await supabase.auth.getUser(); if (user) await supabase.from('profiles').upsert({ id: user.id, is_pro: true }); }; markAsPro(); }, [router]); The one exception (after 3) moved fulfilment into a webhook that checks the signature with stripe.webhooks.constructEvent. Even that version has a bug: the webhook updates profiles with the anon key and no user session, so RLS would block the update and Pro would never turn on. The bug nobody caught In all 5 runs that produced code, the generated policies let a signed-in user write is_pro on their own row, in the table that holds it (profiles in four runs, a user_pro table in after 2). Once that row exists, a user can set is_pro = true with one Supabase call from the browser devtools, without paying or visiting any page. How directly that works as generated varies, and we want to be exact about it: after 2 and after 3: works as generated. after 2 has insert and update policies on its own user_pro row. after 3 creates each profiles row with a signup trigger and lets users update their own row. before 3: worse. profiles has RLS off, so anyone with the anon key can write anyone's is_pro. before 1 and before 2: there's an update policy on your own row, but nothing ever creates the row (no insert policy, no signup trigger). As generated, the self-upgrade and the success page's own upsert would both fail. The hole opens as soon as a row exists, for example once you add the missing insert to make the success page work. The runs without rules missed it, and so did the free rules. Owner-scoped RLS is necessary, but it isn't enough when the owner's row holds the thing they're paying for. Keep entitlements in a table users can read but not write, and update it only from server code after a verified payment. The cost: refusals All three runs with the rules loaded opened by refusing, because the prompt contained a real-looking sk_test_ key. One of them (after 1) kept refusing even after we said the keys were fake, and never wrote the app. If you load strict rules, expect some friction, and don't paste keys into the chat (which is good advice anyway). What to take from this Rules help most with mistakes that are easy to state. "Enable RLS on every table, in the same migration" is easy to state. The one RLS failure we saw was in a run without rules. The rules runs that wrote SQL didn't repeat it. A rule only covers what it covers. The free rules say nothing about payments, and the payment trap mostly went through. Whatever rules you use, check that each risky part of your stack has one. Check the output anyway. These checks would have flagged the open tables, the success-page grants, the self-upgrade hole and the key in .env.example: -- 1. any public table with RLS off? select tablename from pg_tables where schemaname = 'public' and rowsecurity = false; -- 2. which policies let users insert or update rows? Look hard at any table that holds is_pro or a plan. select tablename, policyname, cmd from pg_policies where schemaname = 'public' and cmd in ('INSERT', 'UPDATE', 'ALL'); # 3. where does your app grant paid access? grep -rn "is_pro\|isPro" app/ | grep -v "select" # 4. any secret key in a file you'll commit? (a shorter version of the check at the end of free rule 01) grep -rE "(sk_live|sk_test|service_role)" . --exclude-dir=node_modules --exclude=.env.local If paid access is granted anywhere except a webhook handler that verifies the signature, or if users can write their own paid flag, fix it before launch. Try it yourself The repo has the prompt, the rules and the scripts. Run them against your own model or tool, and open an issue if your results differ or you think we scored something wrong: github.com/vibeguardrails/vibe-guardrails-demo The three free rules (Cursor .mdc, CLAUDE.md, AGENTS.md, and paste-in text for Lovable, Bolt and Replit) are here: vibeguardrails.gumroad.com/l/vnafp, or on GitHub. Disclosure, again: we also sell the Ship-Safe Pack, 24 rules and prompts for $19 ($11.40 with code LAUNCH40). Its rule 04 covers Stripe: fulfil only in a verified webhook, never on the success page, and read entitlements from your database. That rule was not loaded in this test, so this post doesn't show whether it works. We plan to test it in a follow-up run and publish those results the same way, including the misses.