A 200 response is not a page, and your policy check is grepping an empty shell

I run automated checks over terms of service and publisher agreements before we use a platform. Fetch the page, strip the tags, grep for the clauses that matter. It is not sophisticated and it has worked for months. This week it lied to me three times. The numbers Same script, same extraction, three legal pages: peerlist.io/terms returned 60 characters of text. daily.dev/terms returned 70. substack.com/pa, which is a full publisher agreement, returned 1,345. For comparison, the same script on a server-rendered legal page returns between 20,000 and 35,000 characters. Substack's terms of use, fetched the same way, gives 22,045. All three responses were HTTP 200. Nothing errored. Nothing retried. The script reported no matching clauses and moved on. Why this is worse than a 404 Those three pages are client rendered. The HTML that arrives is a shell, and the text I care about gets assembled by JavaScript after load. Old news, and normally a visible problem, because you print the body, you see nothing, and you reach for a browser. What let it past me was the shape of the check rather than the shape of the page. My check is a negative assertion. The pass condition is "this grep found nothing." An empty document satisfies every negative grep ever written, so a page that failed to load produces the same verdict as a page that loaded fine and genuinely has no such clause. The failure is silent, and it resolves toward clean, which is the expensive direction. The Substack case is the one that would have cost me. That publisher agreement does contain a clause I needed to know about. My script pulled 1,345 characters of shell, found no match, and would have told me the document was clean if I had trusted it. The rule Any check whose success condition is absence has to prove the input was present first. That reads as obvious written down. It is not obvious while you are writing the thing, because the presence check is the part nobody writes. You write the grep, the grep works on the first page you try, and the invariant that the page actually loaded stays implicit forever. What I do now Three assertions before the grep runs, and a verdict of unresolved rather than clean when any of them fail. MIN_LEN = 3000 # a real legal page is 20k+, a shell is under 1.5k SENTINELS = ("terms", "agreement", "policy", "last updated") def check_policy(url, patterns): text = extract_text(fetch(url)) if len(text) < MIN_LEN: return "UNRESOLVED", f"only {len(text)} chars, probably a client-rendered shell" if not any(s in text.lower() for s in SENTINELS): return "UNRESOLVED", "no legal-document markers, wrong page or a redirect" hits = [p for p in patterns if re.search(p, text, re.I)] return ("HITS", hits) if hits else ("CLEAN", []) The length floor does most of the work. Pick it from your own corpus rather than from a guess: fetch ten pages you know are fine, take the smallest, halve it. The sentinel check catches the second thing that bit me. You can pull 30,000 characters of perfectly good text off entirely the wrong document, because the URL redirected to a marketing page, or because a 404 rendered as a styled landing page instead of a status code. Unresolved is a real state and it needs somewhere to go. Mine escalates to a headless browser, which costs about a second and settles it. The point is that unresolved never quietly collapses into clean. The second trap: the URL you were given may not be the URL substack.com/publisher-agreement returns 404. The document exists and lives at substack.com/pa. I only found it because the footer link on the terms page pointed there. So stop guessing slugs. Fetch a page you know exists, read its links, and follow the one whose anchor text matches what you want: def resolve_doc(index_url, label): soup = BeautifulSoup(fetch(index_url), "html.parser") for a in soup.find_all("a", href=True): if re.search(label, a.get_text(), re.I): return urljoin(index_url, a["href"]) return None # not found is not the same as not there, so escalate Three lines of real work, and it survives the vendor reorganising their legal pages, which they do more often than you would expect. Where else this shape turns up Status page checks that pass because the incident list never rendered. Sitemap validators reporting zero broken links because the sitemap came back empty. Index freshness checks reporting no stale documents because the query errored and returned an empty set. Anything that counts problems and alerts on a nonzero count will report perfect health the moment its input pipeline dies. The tell is the same every time: the healthy state and the broken state produce identical output. If your monitoring cannot separate "nothing wrong" from "nothing checked", it is not monitoring that thing. The short version Feed your check an empty string once and read what it says. If it says everything is fine, you have found the bug. Written from a week of policy checks that returned 200 and told me nothing. Drafted with AI assistance, verified against the live pages described, and edited by hand.