A Model May Propose a Documentation Example; Only a Human Can Assert What It Prints

The rule I apply is short: every fenced block in a documentation file is either executed by CI or labeled illustrative with a named owner. The model may propose the code in either case, but only a human may assert what that code prints. That single constraint removes the most common defect in model-drafted docs, which is a confident example that has never run. This is a narrower claim than "generate your documentation", and it is deliberately narrower, because the artifact below fits on one screen and blocks a whole class of review arguments. The executor is plain Python with no dependencies, and the gate treats an unlabeled fence as a policy violation rather than a style nit. 1. Why the fence, and not the paragraph, is the unit of trust A wrong sentence in prose costs a reader a minute of confusion, while a wrong command costs them an afternoon of debugging their own environment. Code blocks therefore carry a different burden of proof than the paragraphs around them, and reviewers tend to treat prose and code as one artifact. Separating the two lets you apply human attention where the cost of error is concentrated. Models are genuinely good at the connective prose in a tutorial: the framing sentence, the "now that you have X" transition, the summary of what changed. They are unreliable about the observable consequences of a command, especially for CLIs, SDKs, and config schemas that changed after the training data. That asymmetry, not general quality, is what the gate encodes. The third reason is review economics. A reviewer who is asked to verify six commands by hand will verify two and skim four, and a reviewer who is shown which blocks already ran will spend the saved time on the parts a machine cannot check. Gate first, review second. 2. Label every fence with one of three verbs The convention needs to survive a diff without extra tooling, so the label lives in the fence info string and nowhere else. Label Meaning Who may author it runnable Executed in a throwaway directory; non-zero exit fails the build Model may propose, human must confirm it runs output The stdout/stderr a human actually observed for the runnable block directly above Named human only illustrative owner=@handle A sketch that must never execute: pseudocode, partial config, a truncated JSON payload Model may draft, owner vouches for shape Optional attributes stay minimal and mechanical: timeout=120 when a block is slow, and match=contains when the real output is noisy and only a substring is stable. Resist adding more attributes, because every attribute is a thing a reviewer must interpret under time pressure. A useful side effect is that illustrative forces a decision. When a model invents a plausible curl example and the author cannot run it, the block becomes illustrative owner=@handle and the doc says so implicitly. Readers who copy it know they are on their own. 3. The executor: one file, standard library only The parser walks lines, collects fences, and runs each runnable block in a fresh temporary directory so that one example cannot leak state into the next. #!/usr/bin/env python3 """docex: every fenced block in docs is either executed or explicitly labeled.""" from __future__ import annotations import argparse, json, re, subprocess, sys, tempfile from dataclasses import dataclass from pathlib import Path FENCE = re.compile(r"^``` (?P[\w+-]*)\s*(?P.*)$") ATTR = re.compile(r"(\w+)=(\S+)|(?