A UI test that guesses is worse than one that fails

The worst bug I've seen in a test suite wasn't a failure. It was a pass. A login test tapped a button labeled "Continue". The app then shipped a second "Continue" inside a consent sheet, and the recorded test — which resolved its selector by taking the first match — started tapping the wrong one. It stayed green for two weeks, because the flow still ended up somewhere plausible, and a passing test is not something anyone goes back and reads. So that became the rule for tapflow's flow runner: anywhere the tooling has to choose for you, it has to say so. Finding those places took longer than fixing them — there were four. Last time I wrote about getting an accessibility tree out of a simulator that has no window, on both iOS and Android, in one schema. This post is about what that tree is for. The automation axis here — the flow runner and the MCP server — is experimental. The manual browser QA path is the mature one. The flow file A flow is YAML, replayed by tapflow flow run with zero LLM calls. The same input produces the same steps in the same order, every time: name: login-smoke appId: com.example.app steps: - clearState - launchApp - assertVisible: "Sign in" - tapOn: { id: "com.example.app:id/email" } - inputText: "user@example.com" - tapOn: "Sign in" - assertVisible: { label: "Orders", timeout: 15 } There are ten steps in the entire vocabulary — clearState, launchApp, tapOn, inputText, pressKey, swipe, scroll, openUrl, assertVisible, and assertNotVisible. That's deliberate. Every step we add is another way for a flow to end up meaning something other than what it looks like it says. Ambiguity is a bug in the selector A bare string resolves in a fixed order — exact identifier, then exact label, then partial label — and once one of those stages matches, the later stages aren't tried. If more than one element survives that stage, a tapOn fails on the spot and tells you what it found: 2 elements match "New Orders" — add an index or a more specific role/label (candidates: button "New Orders" | text "New Orders") This is the case from the top of the post. Taking matches[0] would let a suite silently retarget after a redesign, and there's no way to tell that apart from a suite that still works until something downstream breaks. The candidate list also turns a five-minute stare at a screenshot into a one-line edit. Two disambiguators resolve it. role narrows by element kind, which handles the common case where a button and the text inside it carry the same label. index picks the Nth remaining match, zero-based, for rows that have neither a label nor an identifier: - tapOn: { label: "New Orders", role: button } - tapOn: { role: cell, index: 2 } The object form needs at least one of id, label, or role. An index on its own would be a positional coordinate again, just written differently. (assertVisible is checking presence rather than choosing a target, so it passes when at least one element matches.) There is no sleep step Timing was the next place the runner was guessing. A sleep records how slow CI happened to be on the day you wrote the test, and then every later run either wastes that time or fails because the machine was busier. So waiting is always a condition with a deadline instead: assertVisible with a timeout, which defaults to 10 seconds and can be set per selector. That was the design, and it had a hole in it that only showed up in use. Right after launchApp, the app isn't in the foreground yet, so a tree query legitimately throws. The first version of the poll loop treated any thrown query as a step failure, which meant a flow that began with a launch failed immediately. People worked around it by adding a long-press somewhere as a stand-in for a sleep. We had removed sleeps and handed everyone a worse one in return. The loop now separates a failure worth waiting out from one that waiting cannot fix: async function queryOrRetry(driver: FlowDriver, deadline: number) { try { // bound the query by the remaining deadline: a stalled response // must not block the loop past the step's own timeout return { tree: await driver.queryUITree(AbortSignal.timeout(deadline - Date.now())) } } catch (e) { if (e instanceof TransientQueryError) return { transient: e.message } // keep polling throw e // bad request, auth, missing session → fail now } } A foreground race, an idle timeout, or a network blip is retried until the deadline. A malformed request or a missing session fails immediately, because waiting won't fix either one. And when a wait does time out, the last transient error is included in the message, so "no element matched X within 10s" can't hide the fact that every query along the way was erroring. Was that a test failure, or a dead runner? The next one only shows up in CI, at 3am, when a red build tells you something broke but not whom to wake — the person who wrote the checkout screen, or the person who owns the Mac in the closet. So the exit codes are a contract: Code Meaning 0 All flows passed 1 At least one flow failed 2 Environment/config error (parse failure, relay unreachable, no device) - name: Run flows run: | tapflow flow run .tapflow/flows/*.yaml \ --relay "$TAPFLOW_RELAY_URL" \ --device "iPhone 16 Pro" \ --build "$BUILD_ID" \ --junit report.xml --junit writes one testcase per flow, and a failure drops a screenshot from the moment it failed into .tapflow/artifacts/. The launchApp step takes no argument and launches whatever --build installed, which keeps build ids out of the flow file so the same flow can run against a fresh build every time. Where a model is allowed to guess The last one came later, and it's the reason the runner does no inference at all. An agent can drive a session through the MCP tools — tap, type, read the tree — and author a flow out of what it just did. That part is exploratory and non-deterministic, and that's fine. What it produces is then replayed by the runner with no model in the loop, and the replay is what decides whether the flow was any good. Generation is allowed to be fast and occasionally wrong, because the deterministic pass is the gate it has to clear. The same engine backs the MCP run_flow tool, so an agent can author a scenario once and replay it afterward instead of re-deriving it on every run. Replay costs nothing in API calls, which is also why it's cheap enough to run on every commit. What this is really for Flow Capture is the reason any of this exists. A person tests a build by hand in the browser, the way they already do, and every tap is recorded as a selector read from the tree instead of a coordinate. The QA someone already performed becomes the flow that runs on every build after it. It isn't built yet, and I don't want to oversell it. What is done is the part it stands on: the tree, the shared schema, and a runner that reports an ambiguous situation instead of resolving it silently. Limits worth knowing Selector matching is the roughest edge here, especially in the seconds right after a launch, and the automation axis as a whole is experimental. If your app has screens where nothing carries an identifier or a label, role and index will get you through them, but a flow written that way is more brittle than one written against a screen with accessibility identifiers. That's its own argument for adding them. Try it npm install -g tapflow tapflow start tapflow flow run .tapflow/flows/login-smoke.yaml Repo: https://github.com/jo-duchan/tapflow Docs: flow reference · MCP in CI/CD If your suite currently resolves selectors by taking the first match, it's worth going and looking at what it's tapping today.