Approval Testing for Prompt Output

Approval testing is an older idea than snapshot testing and a more honest description of what a golden-file suite for model output does. The test does not know the right answer. A person does, and the file records that they looked. The received and approved pair The mechanism is two files per case. The run writes what it got to a .received file. If a .approved file exists and the two match byte for byte, the received file is deleted and the test passes. If they differ, the test fails and a reporter launches — a diff tool, an editor, or in CI a plain textual diff — showing the human exactly what changed. Approving means copying received over approved and committing it. In ApprovalTests.Python the call is verify(result), imported from approvaltests.approvals, and the file names are derived from the test’s identity. The library exists in Java, .NET, C++ and JavaScript with the same two-file protocol. Nothing about that protocol is specific to language models; it was built for legacy code whose output nobody could specify in advance, which is a strikingly good description of a prompt. # tests/test_summary.py from approvaltests.approvals import verify from app.summarise import summarise_ticket def test_ticket_summary(): verify(summarise_ticket(load_fixture("ticket-4412.json"))) The received file is a build artefact and belongs in .gitignore; the approved file is the asset and belongs in the repository next to the test. Getting that the wrong way round produces a suite that passes locally and fails everywhere else, because each machine approves its own output. Why the framing changes what you build Call it a snapshot test and the implicit claim is that the recorded value is correct, which invites the reflex of pressing the update flag to make red go green. Call it an approval test and the claim is narrower and true: a named human looked at this output on this date and said it was acceptable. That reframing has three consequences that show up in the design. The diff is the product, not a side effect. If a reviewer cannot decide in a couple of minutes whether a change is acceptable, the approved file is formatted wrong. That is an argument for storing a normalised, field-per-line reduction rather than raw prose, and it is the same argument the rest of this cluster makes from the failure side. Batch size is a design parameter. Approval is human attention, and human attention does not scale with corpus size. A suite of eight hundred approved files that all regenerate together cannot be approved; it can only be rubber-stamped. Split the corpus into a small tier that is reviewed every time and a large tier that is checked by invariants instead. Approval has a date and an owner. Because the approved file is committed, git blame answers “who accepted this wording, and when” without any extra machinery. That is the audit trail people build spreadsheets for. What an approved file does and does not claim An approved file is evidence of review, not evidence of correctness. The reviewer saw one output for one input at one moment. They did not see the output for the ninety-nine inputs nobody wrote a case for, and they did not see the second sample from the same prompt, which — since nothing in the approval protocol pins the sampler — may differ. This is the limit that decides where approval testing belongs in a strategy. It is excellent at detecting change and blind to everything that was always wrong. A prompt that has confidently mishandled European date formats since the day it was written will have an approved file recording exactly that, approved by somebody who was checking for a regression and not for a bug. Approval testing pairs with a labelled golden dataset, which encodes what the answer should be independently of what the system produced, and with explicit thinking about blind spots. Who is allowed to approve The question that decides whether any of this survives contact with a team is who may run the approve step. If the answer is “whoever is on the pull request”, then the person who wrote the prompt change approves their own output, and the check is decorative. The two arrangements that work in practice are a code-owners rule on the approved-file directory so a second person must sign, or a domain owner — the support lead, the compliance reviewer — who owns the corpus for their area and whose approval is the release gate. Either way, the approval must be visible in the same review as the change that caused it. An approved file regenerated in a separate, quietly merged commit is an approval nobody gave. Keep the prompt change and its approvals in one pull request, as two commits, and the governance is enforced by the tool everyone already uses. One practical detail decides whether that rule is followed or resented: how long approving takes. If a reviewer must open forty files to approve one prompt change, they will find a way not to. Reducing the approved artefact to a normalised structure, splitting the corpus so that a routine change touches a handful of cases rather than all of them, and ordering the diff so that the cases most likely to have regressed appear first are not cosmetic improvements. They are what makes the governance affordable, and a governance rule nobody can afford is one that gets waived in the week it first matters. Reporter names, option objects and the exact import paths differ between ApprovalTests ports and have changed across versions. Check the ApprovalTests project documentation for the language and version you are using before wiring up a custom reporter. Related Reviewing a Snapshot Diff Before Approving It Updating Golden Files After an Intentional Prompt Change Golden Datasets: Building and Maintaining One