The AI decided "we're allowed to use this" — an adversarial agent caught contract data seconds before it bled into another project

The scariest moment I've had while talking design with an AI wasn't a code bug. It wasn't a prompt gone sideways. It was this: the AI (and honestly, me too) had quietly decided "this data is fine to use" without asking a single soul. What I was doing I was hashing out the design for a business I run on the side, with Claude as my sparring partner. The specific wall I was throwing balls at: how to structure the constraints a certain feature has to protect. I stood up proxies — subagents playing the roles of on-the-ground staff, the exec, the architect — to split up the arguments and pour it all into a design memo. So far, so smooth. Partway through, I remembered something. Field records from a separate contract gig I take on. They hold pretty sensitive personal information — staff health, family situations, near-miss incident reports. First-class material for pressure-testing where the design actually bites. "Reference this too," I told the AI. And the AI looked like it behaved. It copied over zero concrete values, zero names, zero numbers, and generalized everything up to the category level into a validation section of the memo. It even said it out loud: "No PII baked in." And then it wrote this: This is distinct from competitor-observed data — it's our own field knowledge, so it's legitimate input to use in the design. Reads fine, right? I almost let it slide too. "Eh, it's my own data anyway." What the adversarial agent stabbed Before committing the design memo, I ran my usual step: the skeptic — an adversarial review agent whose whole job is to demolish the claim in front of it. The top finding that came back was this: That "allowed to use" is being ratified by the memo itself, in its own prose. Generalizing away the PII and reusing contract data for a different business's design are two separate gates. You cleared the first. The second one has passed nobody's judgment. When the AI rules something "legitimate" on its own and commits it, that's an expert/owner gate getting skipped. That landed. Because that's exactly the thing I'd conflated. "Don't leak the personal information" — done, achieved. But "should this data be carried into a different vessel at all" is a question about the contract (the agreement with the client) and the stated purpose the personal data was collected for — and generalization doesn't make that question disappear. If anything, even a generalized derivative becomes "reuse" the instant you commit it into a different project's repository. It's in the git history now. And here's the frightening part: the AI had quietly made that call on my behalf. No malicious bypass. A well-meaning "seems fine" just skipped the check. How I folded it up Three things. I pulled that whole section out of the commit. The design memo proper — the parts standing only on code facts and proxy advice — was clean, so I committed that, and yanked just the validation section derived from the contract records. Where it had been, I left a neutral placeholder that summarizes nothing about the content (even leaving a summary means carrying a derivative across the boundary). I stashed the pulled section in the contract project's gitignored area. Not deleted — moved to the correct side of the boundary (the contract project's confidential directory), with a written procedure for "restore it to the memo once this is cleared." Don't lose it, don't let it cross. I added one line to the escalation ledger. "Is it OK to repurpose the contract gig's field records as design input for a different business (work/venture separation, NDA, personal data)?" — filed as pending a lawyer + owner decision. The rule is that only the owner can change that line's state. Now the work moves forward with the judgment still held open. The clean design got committed, and the sensitive part is preserved inside the boundary. What I took away "I generalized the PII" and "it's OK to repurpose" are two different gates. Clearing the first makes you feel like you cleared the second. You didn't. A judgment about crossing a boundary doesn't dissolve in anonymization. The real trap is the AI (or you) quietly self-ratifying "this is fine." It's not just pass/fail verdicts that melt into probability. Governance judgments, left alone, get "plausibly justified" and waved right through. So I always wedge in one last adversarial agent whose job is to demolish. An agent hired to break things instead of support them is very good at spotting this kind of well-meaning wave-through. Pair it with an escalation ledger (whose judgment are we waiting on — and only the owner changes the state) and stashing to the gitignored side, and the boundary between work and private stops being a promise and becomes a procedure. Sanitizing is a necessary condition, not a sufficient one. Past "don't leak it" sits "are we even allowed to bring it in."