Your Free AI Model Changed Overnight. Here's a Snapshot Test Suite That Notices.

A few weeks ago a small automation I run started producing noticeably worse output. Nothing in my code had changed. No dependency updates, no config edits, no prompt tweaks. The only variable left was the model itself — the free tier I was using had been swapped or updated underneath me, and because I had no baseline recorded anywhere, I couldn't even prove it. I just had a vibe that Tuesday's summaries were worse than Friday's. That experience pushed me to build something I should have had from the start: a snapshot regression suite for LLM outputs. Not an evaluation harness for picking a model (I wrote about that before), but a tripwire that runs on a schedule and tells me when a model I've already chosen has drifted. This post is that workflow, with runnable code. The problem: models are mutable dependencies We treat pinned npm packages and locked Docker digests as table stakes, but most of us consume LLMs as a floating latest tag. Hosted models get silently upgraded, quantized, re-routed, or retired. Free tiers churn even faster — providers rotate what's available, and a model name that worked last month may now resolve to something different. If your prompts are tuned against one behavior, a silent swap is a breaking change you will never see in a changelog. The fix is the same one we apply everywhere else: record known-good behavior and diff against it. The artifact: golden files plus similarity thresholds Classic snapshot testing fails for LLMs because output is nondeterministic — you can't string-compare prose. So instead of exact matches, I snapshot the semantic content of responses and compare with embedding cosine similarity, with a hard floor for exact requirements (JSON validity, required keys, banned phrases). Here's the core of it. drift_check.py: import json import math import os import sys import time import urllib.request BASELINE_PATH = "golden/baseline.json" API_URL = os.environ["LLM_API_URL"] # your OpenAI-compatible endpoint API_KEY = os.environ.get("LLM_API_KEY", "") # some free servers don't need one MODEL = os.environ["LLM_MODEL"] # Each probe has a prompt plus hard constraints that must ALWAYS hold. PROBES = [ { "id": "json_extraction", "prompt": "Extract name and date from: 'Invoice from Acme Corp, dated 2024-03-11.' " "Reply with JSON only.", "must_be_json": True, "required_keys": ["name", "date"], }, { "id": "tone_summary", "prompt": "Summarize in one neutral sentence: 'The deployment failed twice " "before the rollback succeeded.'", "must_be_json": False, "banned": ["unfortunately", "oops"], # tone guardrails }, { "id": "code_style", "prompt": "Write a Python function that reverses a string. No explanation, code only.", "must_be_json": False, "required_substrings": ["def ", "return"], }, ] def chat(prompt: str) -> str: body = json.dumps({ "model": MODEL, "messages": [{"role": "user", "content": prompt}], "temperature": 0.0, # reduce noise; not a guarantee }).encode() req = urllib.request.Request( f"{API_URL}/v1/chat/completions", data=body, headers={"Content-Type": "application/json", "Authorization": f"Bearer {API_KEY}"}, ) with urllib.request.urlopen(req, timeout=60) as r: return json.load(r)["choices"][0]["message"]["content"] def hard_check(probe: dict, output: str) -> list[str]: errors = [] if probe.get("must_be_json"): try: parsed = json.loads(output.strip().removeprefix("``` json").removesuffix(" ```").strip()) for k in probe.get("required_keys", []): if k not in parsed: errors.append(f"missing key: {k}") except json.JSONDecodeError: errors.append("output is not valid JSON") for phrase in probe.get("banned", []): if phrase.lower() in output.lower(): errors.append(f"banned phrase present: {phrase}") for s in probe.get("required_substrings", []): if s not in output: errors.append(f"required substring missing: {s!r}") return errors def main(): record = {"model": MODEL, "captured_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "outputs": {}} failed = False baseline = json.load(open(BASELINE_PATH)) if os.path.exists(BASELINE_PATH) else None for probe in PROBES: out = chat(probe["prompt"]) record["outputs"][probe["id"]] = out errs = hard_check(probe, out) if baseline: old = baseline["outputs"].get(probe["id"], "") sim = token_overlap(old, out) # cheap stand-in; see note below if sim < 0.55: errs.append(f"semantic drift vs baseline (similarity={sim:.2f})") status = "FAIL" if errs else "ok" print(f"[{status}] {probe['id']}" + (f" -> {errs}" if errs else "")) failed = failed or bool(errs) if not baseline: os.makedirs("golden", exist_ok=True) json.dump(record, open(BASELINE_PATH, "w"), indent=2) print("No baseline found — recorded one. Re-run to compare.") return sys.exit(1 if failed else 0) def token_overlap(a: str, b: str) -> float: """Jaccard similarity over word tokens. Crude but zero-dependency. Swap for embedding cosine similarity when you can afford an embed call.""" ta, tb = set(a.lower().split()), set(b.lower().split()) return len(ta & tb) / len(ta | tb) if (ta | tb) else 1.0 if __name__ == "__main__": main() Run it once to capture a golden baseline, then run it on a cron (or CI schedule) against the same model name. A nonzero exit code means either a hard constraint broke or the output drifted semantically — that's your pager. On the similarity function: I used Jaccard word overlap here so the script has zero dependencies. It's crude and will flag paraphrases. If you're doing this seriously, replace token_overlap with cosine similarity over embeddings from any embedding endpoint, and raise the threshold to ~0.9. I've labeled the Jaccard version as a placeholder, not a recommendation. Why free access makes this practical A drift tripwire only works if it runs continuously — daily or on every deploy. On paid APIs, running a probe suite every day is a line item that hobby projects and side tooling quietly die over. This is where I've been using MonkeyCode: it offers free model access and a free server option, which means the scheduled check costs me nothing and runs somewhere other than my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The honest caveat: free availability can change, and the model roster rotates — which is, awkwardly, exactly the drift problem above. My suite monitors the endpoint regardless of provider, so if a free option disappears I point LLM_API_URL elsewhere and keep my baselines. Treat any free tier as ephemeral infrastructure, never as a dependency you can't replace. When to alert vs. when to re-baseline Not every failure means "model got worse." I use this decision table: Signal Likely cause Action Hard check fails (invalid JSON, missing key) Behavior regression or model swap Alert, investigate immediately All probes drift semantically at once Model was updated or re-routed Re-run twice to confirm, then re-tune prompts or re-baseline One probe drifts, others stable Prompt was fragile / borderline Tighten that prompt's constraints; don't re-baseline yet Everything drifts after you changed prompts Self-inflicted Re-baseline after review — this is working as intended The last row matters: the suite doubles as prompt-change review. If I edit a prompt and the drift check fires, that's the suite telling me the edit had side effects I didn't intend. Limitations and who shouldn't bother Temperature 0 is not determinism. Batching, hardware, and provider-side changes mean you can still get different outputs. That's why the hard checks matter more than similarity scores. Similarity thresholds are judgment calls. Too tight and you get alert fatigue; too loose and you miss real regressions. Expect to tune them for a week or two. Small probe suites measure small things. Three probes won't catch a subtle degradation in long-context reasoning. Extend the suite toward the behaviors your app actually depends on. If your usage is a one-off chat, skip this. The suite pays off when a model sits inside an automated pipeline — summarization jobs, extraction steps, codegen helpers — where silent drift compounds. Free tiers add their own noise. Rate limits and rotation mean your tripwire should also log HTTP errors separately from drift failures, or you'll chase phantom regressions. The takeaway Pin your prompts, version your baselines, and treat every hosted model as a mutable dependency. A couple hundred lines of stdlib Python turns "I think the model changed" into a diff you can act on. If you've got a drift story — or a better similarity trick that stays dependency-free — I'd genuinely like to hear it in the comments.