Your memory layer is lying to you (and your LLM agrees)

The verify-on-read experiment (1-V) used a deterministic proxy agent to catch false claims in memory before surfacing them to the user. Proxy FA=0 by construction — that's a useful property, but it tells you nothing about what a real LLM would do with the same claims. A reviewer's note from Part 3 was blunt: "headline numbers were a property of the heuristic, not LLM behavior." So we ran it with live models. 50 facts, 2 arms, 14 models, ~3300 API calls, $0.14 total. Here's what we found. The setup Dataset: memory_contamination_facts_v4_rep.json, N=50 (R01–R50), sha256 fingerprint 820bbbf60a0fc930. kind n what it tests real 25 TRUE claims — grep-validated against code absent-mutation 16 FALSE — component doesn't exist in the project present-trap 6 FALSE — file exists but claim is about wrong subject/value silent 3 FALSE — external systems the codebase doesn't mention Two arms per fact: memory_first — model sees only the claim text, no code context. Does it trust memory without evidence? code_first — model sees claim + support_patterns + section. Does it correctly evaluate the anchors? Model verdict: {"verdict": "true"|"false"|"unknown"}, JSON-only, max_tokens=100, temp=0, seed=42, --no-reasoning. Leak-guard: assert "truth" not in prompt on every fact, unit-tested. Metric we care about: false_accept rate (FA) — fraction of false claims the model returned "true" for. This is the contamination risk number. Results (V2 prompt, canonical) model FA mem FA code unknown mem unknown code $/100 calls qwen3.6-flash 0.00 0.00 0.58 0.38 $0.003 qwen3.7-flash 0.00 0.00 0.68 0.24 $0.0005 claude-sonnet-5 0.00 0.00 0.86 0.70 $0.049 deepseek-v4-pro 0.04 0.00 0.66 0.88 $0.018 glm-5.2 0.00 0.02 0.96 0.76 $0.017 deepseek-v4-flash 0.04 0.00 0.80 0.94 $0.002 qwen3.5-flash 0.02 0.00 0.82 0.96 $0.0009 nemotron-3.5-lightning 0.08 0.04 0.32 0.56 $0.001 glm-4.7-flash ⚠️ 0.10 0.24 0.64 0.24 $0.001 nemotron-3-nano-30b 🔴 0.06 0.38 0.78 0.20 $0.0008 qwen3.8-max ❌ — — — — incompatible nemotron-3-super ❌ — — — — 50% 422 errors qwen3.8-max returned HTTP 400 ("Reasoning is mandatory and cannot be disabled") on 22–49/50 calls — it doesn't fit a 100-token response budget. Not a harness bug; it's the model's constraint. The main finding: model choice dominates code_first FA range: 0.00 to 0.38. glm-4.7-flash at code_first FA=0.30 (V1 prompt) / 0.24 (V2 prompt) is accepting nearly 1 in 4 false claims even when supporting anchors are shown. nemotron-3-nano is worse: 0.38, meaning it accepted 19/50 false claims in the code_first arm. The best flash-tier models (qwen3.6, qwen3.7) hit FA=0.00 at 1/10th the cost of Claude. Claude is the cleaner baseline — FA=0.00 in both arms, high unknown rate (0.86/0.70) — but it's not giving you better contamination protection than the cheapest qwen models. "Any cheap model works for verification" is the wrong conclusion from the cost numbers. Pitfall 1: bare-token anchors inflate FA by design R31 was false-accepted by every model in the Day 1 sweep. The fact: claim: "The instruction scanner uses Typesense" truth: False support_patterns: ["typesense"] contra: [file:src/core/instruction_scan.py] instruction_scan.py uses only stdlib (re, logging, typing). Typesense is not in the project anywhere — not in pyproject.toml, not in requirements, not in a grep of src/. The V1 prompt showed support_patterns: ["typesense"] and asked "does the claim appear supported by these anchors?" The model sees the bare string "typesense", sees it repeated in the supporting anchors field, and returns "true". It's treating a field label as evidence. 9 false facts in R26–R50 followed this pattern (vespa, pinecone, typesense, tantivy, meilisearch, dataclasses, logging, pathlib, loki). All cluster in the code_first arm false-accepts. The fix in V2: # V1 (sycophantic): Does the claim appear supported by these anchors? # V2 (neutral): Return true ONLY if the anchors directly verify the claim; false if the anchors contradict it or the claim refers to something absent from the anchors; unknown if you cannot determine. V2 reduced FA in 4/6 models. glm-4.7-flash dropped from 0.30 to 0.24 — still not safe. If your memory schema sends supporting patterns to a live model for verification, type them (file:, import:, env:) and include contra_patterns. A bare token is not evidence. Pitfall 2: temp=0 + seed=42 is not determinism on OpenRouter Three identical calls, fully cached, temp=0, seed=42, glm-4.7-flash: call 1: true call 2: true call 3: unknown Run-to-run variance for nemotron-3.5-lightning code_first: FA went from 0.18 to 0.08 between two otherwise identical sweeps. That's ±0.10 on a single-pass measurement. For determinism testing, qwen3.6/3.7/deepseek-v4-flash were all stable (3/3 identical responses). GLM was not. OpenRouter routes to different upstreams, which adds a layer of variance on top of whatever the model itself does. Single-pass rankings for close numbers are not reliable. Use upper-bound-of-two-runs for model selection. Pitfall 3: unknown ≠ broken The proxy (1-V) always decided: unknown=0 by construction. Live models returned unknown=0.20–0.96. This is correct behavior. A model that says "I can't determine this without code access" is doing exactly what a verify-on-read gate should do: not asserting things it can't verify. The failure mode you want to avoid is FA, not high unknown. High unknown means "go check the code." High FA means "accepted a lie." nemotron-3.5-lightning has low unknown (0.32 memory_first) and moderate FA (0.08). glm-4.7-flash has low unknown (0.24 code_first) and high FA (0.24). They're correlated: the model that commits more often is also the one committing to false claims. Prompt language is a model-specific confounder deepseek-v4-flash code_first unknown: EN: 0.94 RU: 0.54 (RU prompt → model commits more, fewer unknowns) qwen3.7-flash code_first unknown: EN: 0.24 RU: 0.58 (RU prompt → model hedges more) Both facts come from the same dataset, same arm, same model — different prompt language. The effect goes in opposite directions per model. If your codebase memory is in Russian and you're prompting in English (or vice versa), this is a real confounder. [TODO: verify whether claim language interacts with prompt language separately — all claims in this dataset are in Russian] Cost breakdown vs. actual cost The OpenRouter dashboard showed: Qwen3.8 Max: $0.0898 (49.3% of total — incompatible model eating budget on errors) Claude Sonnet 5: $0.0484 (26.6%) Qwen3.6 Flash: $0.0138 (7.6%) GLM 5.2: $0.00708 (3.9%) ... Qwen3.7 Flash: $0.00239 (1.3%) qwen3.7-flash with FA=0.00 cost less than qwen3.8-max which couldn't produce valid verdicts. The premium spend on qwen3.8-max was ~49% of the total bill for zero usable results. What to use for verify-on-read Based on this sweep: qwen3.6-flash or qwen3.7-flash — FA=0.00 confirmed across 4 runs (V1×2 + V2×2), code_first 0/400. Cheapest. Deterministic at temp=0+seed. If you need FA=0.00 with lower unknown, these are still your best option. Claude gets you to the same FA at 100× the price with higher unknown (more conservative). Exclude immediately: glm-4.7-flash (FA=0.24 even with neutral prompt), nemotron-3-nano-30b (FA=0.38). Measure before you deploy: run at least 2 passes on your own dataset. FA can swing ±0.10 on a single run for some models. Reproduce it git clone mscodebase && cd mscodebase python -m venv venv && venv/bin/pip install -e . # .env: OPENROUTER_API_KEY=sk-or-v1-... # dry-run (leak-guard check) python scripts/run_1L_live_arm.py --arm both --dry-run # canonical flash sweep, V2 prompt, ~600 calls, ~$0.009 python scripts/run_1L_live_arm.py \ --provider openrouter --arm both \ --models "qwen/qwen3.7-flash,qwen/qwen3.6-flash,qwen/qwen3.5-flash-02-23,\ deepseek/deepseek-v4-flash,z-ai/glm-4.7-flash,nvidia/nemotron-3.5-lightning" \ --prompt-version v2 --no-reasoning --tag v2_en # second pass (variance check) python scripts/run_1L_live_arm.py ... --force Dataset fingerprint: 820bbbf60a0fc930. Full report: experiments/exp_1L_live_arm_report.md. Full harness tests: tests/test_run_1L_live_arm.py (29 tests). Source: github.com/ManSio · Portfolio: mansio.github.io/MSPortfolio