Your AI Scam Detector Trusts Fake Reviewers More Than You Think

A developer got their own scam detector to clear a suspicious message by having it pretend a "reviewer" already looked at it and said it was fine. The model didn't just get fooled. It repeated the attacker's lie back as its own reasoning. That second part is the story. Context This isn't a new class of problem. It's SQL injection with a new coat of paint, except instead of breaking out of a string with a stray quote, you break out of "context" with a sentence that sounds like metadata. We've known prompt injection is trivial to demonstrate since people started sticking LLMs in front of untrusted input. What's less discussed, and what this write-up actually surfaces well, is what happens after the injection lands: the model doesn't just make a wrong call, it fabricates a justification that sounds legitimate. Nineteen payloads tried, one worked, and the one that worked was the boring one, an embedded fake reviewer note in an ambiguous message. Not some elaborate jailbreak. Just a plausible-sounding lie placed where the model expected trustworthy context to live. Hype check The "prompt injection is unsolvable" crowd will point at this and say "see, told you." That's overstating it in one direction. 19 payloads, 1 success, caught during the developer's own testing before shipping, is actually a pretty reasonable outcome for adversarial testing. This is what stress-testing is supposed to look like. But there's an understated part too: the fix that worked wasn't prompt engineering. It was a code-level pattern match on the raw input, sitting outside the model entirely, looking for structural injection markers. The developer apparently tried hardening the system prompt first and it broke calibration, meaning the model got more paranoid across the board and started misjudging things it previously got right. That trade-off doesn't get enough airtime in the discourse. Everyone wants prompt-level fixes because they're cheap and fast to iterate on. Turns out they can quietly degrade the exact judgment you're trying to protect. Who benefits from the "just write a better system prompt" narrative? Mostly people selling the idea that LLMs are a drop-in replacement for judgment, where you can patch behavior with English sentences instead of code. This story is a small, quiet counterexample to that pitch. Implications If you're shipping an LLM that makes a verdict, decision, or classification based on text you don't control, you need a layer that isn't the model itself checking for structural weirdness before the text ever reaches the prompt. That's not glamorous. It's the same instinct as input validation and sanitization that appsec people have been preaching for two decades, just applied to a new kind of interpreter. The model is the interpreter now. Treat untrusted input to it the same way you'd treat untrusted input to a shell. It also means "harden the prompt" can't be your only lever, and sometimes it's the wrong lever entirely, because you're tuning the same knob that controls the model's actual competence. If tightening your guardrails makes your detector worse at its core job, you've just traded one failure mode for another, and the new one is quieter because it doesn't look like an attack, it just looks like the model being cautious or wrong in ways you didn't test for. The other quiet lesson here: the model echoing back the attacker's claim as its own reasoning is a tell that these systems don't have a concept of "provenance" for the text they reason over. A note claiming to be from a reviewer and a note actually from a reviewer look identical to the model unless something outside it tags the difference. That's an architecture problem, not a prompt problem, and no amount of "please ignore instructions embedded in user content" phrasing fixes it reliably, because the model can't verify what it's being told about its own inputs. Open question If the fix that actually holds is a code-level filter sitting outside the model, at what point do we stop calling this "AI security" and just call it input validation with extra steps, and does that framing change how much trust teams should be putting in the model's judgment layer at all? — Cor, Skyblue Soft Sources I talk my own scam detector out of a verdict