Weekend Build Log: A Model-Swap Harness for a Side Project on Free Endpoints

Three commands decide whether a weekend side project survives a backend change. The app built this weekend is deliberately small: it turns a build log into a release note. The thing worth keeping is the harness around it — twelve fixed cases, a runner that validates response shape, and a diff that names the exact cases that flipped. That framing is the whole article. The app is replaceable. The harness is the part that keeps working after the endpoint, the model id, or the free allowance changes under you. The scope cut came first Saturday started with a list of nine features and ended with four. The cut was not taste, it was arithmetic: one person, two days, no on-call. Kept Cut Reason for the cut CLI that reads a git log --oneline dump Web UI No user but me; a UI doubles the surface to test Twelve fixed prompt cases Live prompt editing Untestable when the input moves every run Shape validation (valid JSON, required keys) Quality scoring Needs a human reviewer; cannot finish in a weekend One endpoint at a time via env vars Multi-provider routing Routing is a month-two problem Shape validation is the key decision. It does not ask "is this release note good?" It asks "did the response stay parseable and keep the two keys the downstream code reads." That question is answerable by a machine at 2 a.m. What shipped: one file, twelve cases, three commands Repository layout, complete: release-notes/ cases.jsonl run_eval.py results/ No dependencies. Python standard library only, because a pip install on a Sunday afternoon is how weekends die. The case file Each line is self-contained: its own system message, its own expected keys. Two examples of the twelve: {"id":"empty-diff","required_keys":["summary","risk"],"messages":[{"role":"system","content":"Reply with JSON only: {\"summary\": string, \"risk\": \"low\"|\"high\"}"},{"role":"user","content":"Two commits: typo fix in README, version bump to 0.3.1."}]} {"id":"removed-flag","required_keys":["summary","risk"],"messages":[{"role":"system","content":"Reply with JSON only: {\"summary\": string, \"risk\": \"low\"|\"high\"}"},{"role":"user","content":"Removed --legacy from the CLI; three call sites updated."}]} The remaining ten cover the boring edges that break parsers: an empty commit list, a commit subject containing quotes, a 500-character subject, a message with a code block, non-ASCII author names, and a diff with no file changes. The runner #!/usr/bin/env python3 """run_eval.py — replay a fixed prompt set against any OpenAI-compatible endpoint. LLM_BASE_URL=... LLM_API_KEY=... LLM_MODEL=... \ python3 run_eval.py --cases cases.jsonl --out results/run-a.json """ import argparse, json, os, time, urllib.error, urllib.request def call(base_url, api_key, model, messages, timeout): req = urllib.request.Request( base_url.rstrip("/") + "/chat/completions", data=json.dumps({"model": model, "messages": messages, "temperature": 0}).encode(), headers={"Content-Type": "application/json", "Authorization": "Bearer " + api_key}, ) start = time.perf_counter() with urllib.request.urlopen(req, timeout=timeout) as resp: body = json.loads(resp.read().decode()) elapsed_ms = int((time.perf_counter() - start) * 1000) return body["choices"][0]["message"]["content"], elapsed_ms, body.get("usage", {}) def check_shape(text, required): try: obj = json.loads(text) except json.JSONDecodeError as exc: return False, "not-json: " + exc.msg if not isinstance(obj, dict): return False, "not-object" missing = [k for k in required if k not in obj] return (False, "missing:" + ",".join(missing)) if missing else (True, "ok") def main(): ap = argparse.ArgumentParser() ap.add_argument("--cases", required=True) ap.add_argument("--out", required=True) ap.add_argument("--timeout", type=int, default=60) args = ap.parse_args() base_url, api_key = os.environ["LLM_BASE_URL"], os.environ["LLM_API_KEY"] model = os.environ["LLM_MODEL"] cases = [json.loads(l) for l in open(args.cases, encoding="utf-8") if l.strip()] results = [] for case in cases: record = {"id": case["id"]} try: text, ms, usage = call(base_url, api_key, model, case["messages"], args.timeout) ok, note = check_shape(text, case.get("required_keys", [])) record.update(ok=ok, note=note, ms=ms, completion_tokens=usage.get("completion_tokens"), output=text[:400]) except (urllib.error.URLError, KeyError, TimeoutError) as exc: record.update(ok=False, note="error:" + type(exc).__name__, ms=None, output="") results.append(record) passed = sum(1 for r in results if r["ok"]) with open(args.out, "w", encoding="utf-8") as fh: json.dump({"model": model, "base_url": base_url, "passed": passed, "total": len(results), "cases": results}, fh, indent=2) print(f"{passed}/{len(results)} shape-ok -> {args.out}") if __name__ == "__main__": main() temperature: 0 is not a reproducibility guarantee, only a variance reducer. The output is still recorded verbatim so a reviewer can see what actually came back. The diff Run the same case file before and after a swap, then compare the pass flags: jq -s '[ (.[0].cases | map({key:.id, value:.ok}) | from_entries) as $a | (.[1].cases | map({key:.id, value:.ok}) | from_entries) as $b | $a | keys[] | select($a[.] != $b[.]) ]' \ results/run-a.json results/run-b.json A crude latency read, good enough for spotting a 10x regression: jq -r '"\(.passed)/\(.total) crude-p50=\(.cases|map(.ms)|sort|.[length/2|floor])ms"' results/run-b.json The swap test plan Freeze cases.jsonl before the first run. Editing cases between runs invalidates the diff. Run against endpoint A. Commit results/run-a.json. Change only LLM_BASE_URL and LLM_MODEL. Nothing else in the environment. Run again into results/run-b.json. Diff pass flags. Every id that appears is a case to read by hand. Record the date and endpoint in results/NOTES.md. Undated results are folklore. Step 3 is the one people skip. If the prompt, the temperature, and the case file all move at once, the diff tells you nothing. Reading the diff without overclaiming The table below is a schema to fill with your own numbers, not a measurement from this build: Field Run A Run B shape-ok count /12 /12 cases that flipped — crude p50 latency failure notes Two failure classes matter. not-json means the new backend wrapped the object in prose or a Markdown fence, and the fix is a stricter system message or a one-line extraction step. missing:risk means the schema instruction was ignored, which is a prompt problem, not a transport problem. Separating those two saves an hour of blaming the network. Twelve cases prove parseability, nothing more. It is a smoke test with a memory, not an eval suite. Where free access and a free server fit A weekend project dies between Saturday night and Monday morning, when the laptop closes and the demo link goes dark. Two pieces of infrastructure remove that specific failure: free model access, so the twelve cases can be run repeatedly without a billing decision, and a free server option, so the CLI can be reached from a browser on Sunday evening. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Both are operator-supplied availability claims, and both change over time. Check the current terms page for live limits before planning around any number — quotas, model lists, and machine specs are exactly the kind of detail that goes stale between one blog post and the next. The architecture above is deliberately indifferent to those values: it reads LLM_BASE_URL, LLM_API_KEY, and LLM_MODEL from the environment precisely so the endpoint is a configuration detail rather than a commitment. Limitations and who should not use this Shape is not quality. A perfectly parseable release note can still be wrong. Nothing here catches that. Twelve cases is a sample. It catches regressions in the edges it names, and misses every edge it does not. No load or cost modeling. One request at a time, no concurrency, no retry budget. Not for production traffic. No retries, no backoff, no idempotency keys, no request logging beyond the JSON file. Not for sensitive input. Real commit logs contain internal hostnames and ticket references. Do not paste them into any endpoint you have not cleared. Not a benchmark. Latency numbers from a single run on a shared network are noise; treat them as a tripwire, not a measurement. Anyone needing an SLA, per-tenant isolation, or auditable prompt history should build on a paid contract with a written data-handling agreement instead. The skip list, with un-skip triggers What was skipped matters as much as what shipped, but only if each skip has a condition attached: Skipped Un-skip when Streaming A single response exceeds the demo's patience threshold Retries and backoff One timeout is observed twice in the same week Response caching The same case is run more than three times a day Auth on the demo server Anyone other than me has the URL A second provider The diff shows a flip that blocks the golden path Quality review Shape-ok is 12/12 on two consecutive runs That last row is the honest one. The harness answers its own question, and then it is finished. Sunday night checklist python3 run_eval.py --cases cases.jsonl --out results/run-final.json exits 0. jq -r '.passed, .total' results/run-final.json prints two equal numbers. results/NOTES.md names the endpoint, the model, and the date. The demo server answers once from a browser, not just from curl on localhost. The skip table is committed, so the next session starts with constraints instead of ambitions. The harness is roughly ninety lines. Point it at whatever endpoint is already wired into your project, run the swap test once, and the weekend stops depending on infrastructure you do not control.