Your Automated Tests Are Passing. Here's What They Might Be Missing.

There's a dangerous assumption in software development that doesn't get questioned nearly enough. If the automated tests are green, the application must be working. I've seen teams spend months building test suites, celebrating their coverage numbers, and adding more tests to their CI pipelines. Then a customer reports something that should have been caught weeks ago. The tests weren't necessarily wrong. They were just testing a narrower version of reality. And that's a much harder problem to solve than a broken assertion. The good news is that you don't necessarily need hundreds of additional tests. Sometimes you just need to rethink what you're actually testing. Here are five areas where I think development teams tend to underestimate the problem. 1. Your UI Tests Might Be Ignoring How People Actually Use the Browser Most browser automation revolves around a familiar sequence: Find an element. Click it. Enter some text. Verify the result. That works for a lot of applications. But actual users don't always interact with applications that way. They use Tab to navigate forms. They copy values to the clipboard. They open dialogs, close them, and expect keyboard focus to return to the right place. A test can verify that a modal opens and closes successfully while completely missing the fact that keyboard navigation becomes unusable afterward. And here's where things get interesting. Trying to test these interactions can introduce flakiness if you're checking visual indicators or relying on arbitrary delays. A better approach is to verify the underlying browser state. Instead of checking whether an input looks focused, check whether it actually owns focus. Instead of trusting a "Copied!" notification, verify the clipboard contents. There's a useful technical walkthrough on testing clipboard interactions, keyboard focus, and tab order without creating flaky assertions. The broader lesson is that automating mouse clicks isn't the same as testing the user experience. Especially when accessibility matters. 2. A Fresh Browser Session Can Hide Real Bugs Here's a scenario that probably sounds familiar. A Playwright test passes locally. It passes in CI. Everything looks fine. Then someone opens the application, refreshes the page, and encounters a bug. You try reproducing it in a fresh browser session. Nothing. The temptation is to assume the user did something unusual. But the real problem might be that your tests are starting with a clean browser environment every time. Think about everything that can survive a page refresh: Browser cache Service workers IndexedDB Local storage Authentication cookies These aren't obscure browser features. Modern web applications depend on them constantly. And they can create a completely different experience for someone who's been using your application for two weeks compared with someone opening it for the first time. This is particularly relevant for single-page applications, offline-capable applications, and products with complex client-side state. The SDET published a practical guide on debugging Playwright failures caused by browser state that survives a refresh. One takeaway I particularly agree with is that you need to distinguish between the different persistence layers before trying to fix the test. Clearing everything between executions might make your automation more predictable. But it can also hide the exact bug your customers are experiencing. You need both kinds of tests: isolated tests that start clean, and tests that deliberately preserve state. 3. Passing CI Tests Don't Tell You Whether Your Application Works Right Now Most teams think about automated testing primarily as something that happens before deployment. A developer pushes code. Tests execute. If everything passes, the changes can be deployed. That's useful. But what happens six hours later? Maybe an external API starts returning errors. Maybe an authentication provider becomes unavailable. Maybe a configuration change breaks checkout without anyone deploying a new version of the application. Your CI pipeline won't necessarily tell you about any of these problems. This is where synthetic monitoring becomes useful. Instead of executing tests only when code changes, you continuously run a small number of important checks against your application. For example: Can someone log in? Can a customer complete checkout? Does the API return the expected response? Can users access their dashboards? Of course, the tricky part is deciding what to monitor and how to handle failures. You don't want a monitoring system that wakes someone up because a button took an extra second to appear. You want actionable alerts that represent genuine problems. If you're evaluating this approach, there's a solid breakdown of how to select synthetic monitoring tools for API checks, browser journeys, and alert routing. I think the distinction between automated testing and synthetic monitoring deserves more attention. One helps you decide whether a release is safe. The other helps you discover when something that was previously working has stopped working. You probably need both. 4. AI Applications Introduce an Entirely Different Category of Testing Problems Testing traditional software is relatively straightforward, at least conceptually. You provide an input. You expect an output. Testing applications that use LLMs changes that equation. The same request can produce slightly different responses. The application might retrieve information from different documents. A prompt update might change the response even though the user-facing functionality appears unchanged. And then there's caching. Many applications cache LLM responses to reduce latency and costs. That makes sense economically, especially when inference costs become meaningful at scale. But caching introduces an interesting problem. Imagine you update a system prompt to fix an incorrect response. You run your tests. The application still returns the old answer. Did the model regress? Did the prompt change fail? Is the retrieval system providing outdated information? Or did your application simply return a cached response generated before the update? These are fundamentally different problems. And treating all of them as "AI quality issues" is going to make debugging unnecessarily expensive. A good starting point is this guide on testing LLM output caching without confusing cache hits with model regressions. The important idea is to test the individual layers separately. Verify that caching works correctly. Verify that cached entries are invalidated when necessary. Verify that requests bypassing the cache use the expected prompt, retrieval data, and model configuration. Only then should you start investigating whether the model itself is producing worse results. As more teams add AI functionality to otherwise traditional web applications, I expect this distinction to become increasingly important. 5. Choosing the Wrong Testing Tool Can Be More Expensive Than Writing the Wrong Tests Here's something I think a lot of teams get backward. They spend weeks comparing test automation tools based on features. Does it support Playwright? Does it have AI? Can it test APIs? Does it support mobile applications? Those questions matter. But they're not the questions I'd start with. I'd ask: How much work will it take to maintain these tests six months from now? Who will be responsible for fixing them? Can someone who didn't originally create a test understand what it's doing? And what happens when we need to expand coverage beyond our original requirements? Consider the difference between a platform optimized for broad QA coverage and one optimized for quickly creating readable browser tests. Neither approach is automatically better. For example, this comparison of Katalon and testRigor looks at how those platforms differ in authoring, maintenance, coverage, and governance. The interesting part isn't necessarily which product wins. It's how the tradeoffs change depending on the team using it. A small team trying to automate ten critical browser journeys has very different requirements from an enterprise maintaining thousands of tests across web, mobile, and API environments. There's also an uncomfortable truth about automation. The cost of creating a test is usually only a fraction of the cost of owning it. A test that takes five minutes to create but requires regular debugging and maintenance isn't necessarily cheaper than one that takes longer to create but rarely needs attention. This becomes especially important as AI makes generating new tests easier. Creating more tests is no longer the hard part. Keeping them useful is. So, What Should You Actually Do Differently? I wouldn't respond to these problems by immediately adding another testing framework or doubling the number of automated tests. I'd start with something much simpler. Pick five of the most important user journeys in your application. Then ask yourself a few questions. Are we testing what users actually do? Not just whether buttons work, but whether keyboard navigation, clipboard interactions, and other browser behaviors work correctly. Are we testing realistic application state? A fresh browser session is useful, but it doesn't represent every user. Make sure you're testing what happens after refreshes, repeated visits, and persisted sessions. Are we testing what happens after deployment? Passing CI checks only tells you about one moment in time. Your application needs to keep working long after the deployment finishes. Are we isolating the causes of AI failures? If your application uses LLMs, separate caching, retrieval, prompt behavior, and model output quality. Otherwise, you'll spend too much time debugging the wrong component. Are we optimizing for maintenance, not just creation? The best automation strategy is one your team can actually maintain. And that brings me to the point I think matters most. More Automated Tests Don't Automatically Mean Better Testing It's easy to measure the number of automated tests. It's easy to track how many executions passed. It's even relatively easy to measure code coverage. What's harder to measure is whether your tests are actually reducing the risk of something breaking for your customers. That's the metric worth caring about. I'd rather have 100 reliable tests covering the most important real-world scenarios than 2,000 tests that mostly verify the same happy paths in perfectly clean browser sessions. The goal isn't to automate everything. It's to find the problems that matter, catch them early, and spend as little time as possible maintaining the machinery that does it. That's what good test automation should accomplish. Everything else is just activity. What kinds of bugs have made it into production despite your automated tests passing? I'm particularly interested in cases where the tests were technically correct but missed something important.