Building AI Website Audits with Python: 5 Lessons About Reliability

Building an automated website audit sounds straightforward: fetch a page, run checks, and generate a report. But creating an audit that developers can actually trust is a different challenge. While working with Python, FastAPI, Playwright, and automated website analysis, I’ve been exploring an important question: How do we distinguish a real website problem from something our audit simply couldn’t verify? Here are five engineering principles that make automated audits more reliable. Separate failures from unknown results A common mistake is treating every unsuccessful check as a website failure. For example, a request timeout does not prove that a website is broken. An audit should distinguish between: pass: The requirement was verified. fail: Evidence confirms that the requirement is not met. not_tested: The audit could not complete the check. This distinction prevents unreliable conclusions. Keep deterministic checks deterministic Not everything needs an LLM. HTTP response codes, HTML elements, redirects, and response headers can be checked using conventional Python code. import requests def check_robots_txt(domain: str): url = f"https://{domain}/robots.txt" try: response = requests.get(url, timeout=10) except requests.RequestException: return {"status": "not_tested"} if response.status_code == 200: return {"status": "pass"} if response.status_code == 404: return {"status": "fail"} return {"status": "not_tested"} This example only checks whether robots.txt is accessible. It does not validate the file’s content. For subjective assessments, LLMs can help interpret evidence, but they should not invent missing observations. Require evidence for every finding A useful finding should answer three questions: What was checked? What was observed? Why does the observation justify the result? Instead of reporting: Your website is not AI-ready. An audit should identify a specific, verifiable condition and explain its relevance. Test against unfamiliar websites A system can appear highly accurate when developers repeatedly test it against the same websites. This creates the risk of optimizing for known examples rather than general reliability. A better approach is to test against previously unseen websites, classify false positives, and document the results before changing the implementation. Treat uncertainty as an engineering feature A trustworthy audit does not need to produce an answer for every check. Sometimes the correct answer is: Not enough evidence to determine the result. That is more useful than a confident but unsupported warning. Final thoughts The challenge in AI-assisted auditing is not generating more findings. It is making sure those findings are reproducible, explainable, and supported by evidence. Reliable automation should make uncertainty visible rather than hide it behind a score. ⸻ About the author I’m Ozan Orhan, an AI Automation Engineer based in Hannover, Germany. I work with Python, AI agents, APIs, RAG, MCP, and workflow automation. My technical projects and case studies: ozan-orhan.com