Four false positives in one evening: telling a broken web app from a broken measurement

I spent an evening opening other companies' product configurators — 3D and parametric tools on manufacturers' sites — looking for things that were genuinely broken. Twenty-seven of them. The findings were real. But the part worth writing down is that four separate times in one evening, my tooling told me an application was broken when it was fine. Every one of those four passed automated checks that looked rigorous. What caught them was a screenshot. If you write scripts that judge pages you don't own — uptime checks, competitor teardowns, scraping health, QA of an embedded widget — you will hit these. Here is the full list of signals that lied to me, and the one control that never has. The four false positives All four produced the same symptom: no on the page, and an almost empty innerText. That looks damning when the page is literally titled "Configurator". It is also what three completely healthy situations look like: The tool starts on a click. An orange button launches it. My script measured an unopened door and reported an empty room. Four automated passes — raw HTTP with a browser UA, my own browser, two runs from a clean profile, a control on the same domain — all four confidently examined a page that hadn't started yet. The entire UI lives inside the canvas. One hall configurator draws its menus, its undo/redo and its PDF export in WebGL. Empty DOM text is correct there, not a defect. The tool is behind a login. I was measuring a sign-in page. Fifty-four characters of text and one button reading "Anmelden". The page is a landing page about the configurator, not the configurator. No network-level or DOM-level check distinguishes these from an actual failure. A screenshot distinguishes all four instantly. So the first rule I now follow, before any measurement at all: Take the screenshot first. Look at the picture. What you cannot see in the image, you do not measure. It costs one second and it is the highest-yield step in the whole process. The corollary is that automation answers "is it reachable", never "does it work." When you catch yourself designing a fifth automated pass, what you actually need is one human opening the page and clicking the obvious thing. Signals that lied, and why "No canvas" and "empty innerText". The two least reliable signals I use. Demoted permanently: they are now a reason to look at a screenshot, never a finding. Reading pixels off someone else's WebGL canvas. I nearly built a proof-of-emptiness on this. Copy the canvas into a 2D context, count non-transparent pixels, get zero — on a perfectly live scene. WebGL clears the drawing buffer after compositing unless preserveDrawingBuffer is set. The zero is an artefact of the spec, not evidence about their app. ERR_ABORTED, even on the company's own domain. One site showed eight failed requests, its own domain, its own scripts. That passed my "only count first-party failures" filter and looked like the real thing. Re-run waiting for networkidle instead of domcontentloaded: sixty-five responses, all 200, zero errors. My own measurement was cutting the page off mid-load. A failure is a 4xx or 5xx status in a response — not a request that got aborted, and you must wait for the network to settle before you believe anything. A third-party widget failing. Their chat widget threw net::ERR_FAILED plus CORS. Real, reproducible — and it did exactly the same thing on two unrelated sites, in different countries, on different accounts. That is my environment being filtered, not their code. Same for analytics, tag managers and captcha endpoints: when those fail, you have measured your own ad blocker. transferSize from a persistent browser profile. Returns zero or garbage, because the response came from cache. Weight gets measured from a clean profile or not at all. 403 with no User-Agent. One door configurator returned 403 and 118 bytes to a headless request, and 200 to the same request with a real browser UA. The site was healthy; my headless client was being filtered, and I almost filed it as broken. Without a genuine User-Agent, every 403, timeout and empty body you collect is worthless. Likewise 403/503 from a WAF: that is a block against a datacentre IP, not a defect. A 300×150 canvas. That is the HTML default and proves nothing. Single beautiful observations that didn't reproduce. I saw requests to a URL containing a literal undefined/undefined. Gorgeous finding. It did not appear on either re-run, so it does not exist. The control that always works Everything above is one question wearing different costumes: am I being filtered? For a while I answered it with external text proxies — fetch the same URL through a service on another network, compare. Then all four services I used went down at once: 403, 522, 521, 429. Which taught me the more useful lesson — I ran a known-good file through the same proxies and got the same 521 back. The proxies were down, not the sites. An error from your verification tool is not evidence about your target. The replacement is better than what it replaced, needs nothing external, and works every time: Fetch the neighbouring object, from your own network, in the same request. A 404 on a file? Request the file next to it in the same directory. An empty page? Request a different page on the same domain. A dead domain? Request their homepage. The logic is airtight. Geo-filters, datacentre blocks and User-Agent screening kill the whole connection. None of them can selectively 404 one filename and serve its neighbour. A worked example. A decor texture returned 404 at 564 bytes. Its neighbour in the same directory, same User-Agent, same client, same second, returned 200 at 1877 bytes. No filtering mechanism in existence produces that. The file is genuinely missing. Another: a site whose configurator subdomain returned 200 with a 31-byte body — the entire payload being an HTML comment reading — while its main domain served me full pages normally. For JavaScript errors specifically, note that a text proxy is useless in principle: it doesn't execute scripts, so it can only confirm HTML delivery. There, reproduction from a clean profile plus a control page on the same domain is what you have — or a human on a different connection, whose home ISP is more convincing than any proxy. What survives all of it These findings don't depend on clicks, environments or filters: A 4xx/5xx status from the company's own domain, with a neighbouring object returning 200. A response body too small to contain anything — 31 bytes has nothing to run. No TCP connection at all. My strongest confirmed case: curl returns code 000 and zero bytes, my browser times out at 45 seconds, an external network reports ERR_ADDRESS_UNREACHABLE, and a direct TCP connection to their IP on ports 80 and 443 refuses. DNS resolves; there is no machine behind it. Meanwhile their main site is healthy and links that tool from the menu three times. A library version read out of their own served file. The checklist Screenshot first. If you can't see it, stop. Launch the interactive thing. Click the button. An unopened tool measures as a broken one. Real User-Agent on every request, always. Wait for networkidle. Count response statuses, not aborted requests. Reproduce twice, from a clean profile. Run the neighbour control. No control, no finding. First-party domain only. Their scripts, their responses. Anything that fails a step gets dropped without regret. One operational note that cost me an evening: give every site its own tab. A single timeout leaves a navigation hanging, and every subsequent goto fails with "interrupted by another navigation". Eight sites I had recorded as dead turned out to be one stuck window. The reason this discipline is worth the trouble: I use these findings to start conversations with the companies that own the tools. A message that says "your configurator throws thirteen unhandled exceptions on every load, here they are, I checked twice and from a second network" is worth a great deal — and a message that says it about a tool which is actually fine closes that door permanently and poisons every other message you send. The rigour isn't perfectionism. It's the thing that makes the finding usable at all. I build the kind of tool I was auditing — parametric product configurators and engineering calculators for the browser, where the geometry and the costing are computed from the same parameters so they can't drift apart. Work at smirnov-artur.github.io/webgl, and I'm reachable on Telegram at @smirnovarturr or at paladei702@gmail.com. If you audit pages you don't own, I'd like to hear which signal fooled you — I suspect the list above is incomplete in ways I can't see from where I'm standing.