I summed CPU time by user agent for 7 days: 63% went to clients that never load a second page

The invoice went up around 38% in one quarter and the meeting already had a culprit: the AI feature that shipped in April. Two more instances were on the table. Before approving them, I asked for something cheaper than a server, which was read access to seven days of access logs. The answer took an afternoon. About 63% of the summed backend time in that week went to clients that arrived with a Chrome user agent, requested only HTML, never fetched a single stylesheet, and never came back to the same address twice. Here is exactly what I ran, including the two things that did not work. Your log format is probably missing the only field that matters Most nginx setups still run the default combined format, which gives you status, bytes and user agent. It does not give you time. Without time you can only count hits, and hit count is the metric that hides this problem, because the cheap route wins on volume and the expensive one hides at the bottom of the list. log_format timed '$remote_addr $host "$request" $status ' '$body_bytes_sent $request_time $upstream_response_time ' '$upstream_cache_status "$http_user_agent"'; access_log /var/log/nginx/access.log timed; One reload and you start collecting. Everything below assumes that format. First pass: classify by user agent, and watch it fail The obvious first cut is by declared identity. awk -F'"' ' { split($3, m, " ") ua = tolower($4) if (ua ~ /bot|crawl|spider|slurp/) k = "declared-bot" else if (ua ~ /mozilla\/5\.0/) k = "browser-ua" else k = "other" hits[k]++; secs[k] += m[4] } END { for (k in hits) printf "%-13s %9d hits %11.1f s\n", k, hits[k], secs[k] } ' access.log-2026090* declared-bot 412918 hits 38140.6 s browser-ua 9130477 hits 201773.4 s other 286042 hits 11962.0 s Read that and you conclude the declared bots cost you about 15% of the time and the rest is customers. That conclusion is wrong, and it is the reason most teams stop here and buy the instance. A browser user agent is a string. Anyone can send it, and large scale collection stopped identifying itself a while ago. Second pass: classify by behaviour instead A browser that renders a page also asks for the CSS, the JS bundle, the fonts and a few images. A collector asks for the HTML and leaves. That difference is in the log already, and it does not require a new tool. awk -F'"' ' { ip = $1; sub(/ .*/, "", ip) split($2, r, " "); u = r[2] split($3, m, " ") if (u ~ /\.(css|js|woff2?|png|svg|jpg|ico)(\?|$)/) asset[ip]++; else page[ip]++ secs[ip] += m[4] } END { for (i in page) if (page[i] > 50 && asset[i] / page[i] < 0.1) printf "%-15s %7d pages %6d assets %9.1f s\n", i, page[i], asset[i], secs[i] } ' access.log-2026090* | sort -k6 -nr | head 198.51.100.24 41207 pages 18 14822.9 s 203.0.113.91 28644 pages 0 10310.4 s 198.51.100.77 19855 pages 3 7412.0 s 203.0.113.140 12038 pages 0 4488.7 s ... Sum that bucket against the total and you get the number from the title. In this system it came out at 63.4% of all backend seconds in the week. None of those addresses appeared in more than a couple of hundred requests each, because the ranges rotate. There was no name to block, which is the part that makes the usual answer useless. Third pass: the twenty most expensive routes, by time and not by hits awk -F'"' ' { split($2, r, " "); u = r[2]; sub(/\?.*/, "", u) split($3, m, " ") secs[u] += m[4]; n[u]++ if (m[5] == "HIT") hit[u]++ } END { for (u in secs) printf "%10.1f s %9d %6.1f%% %s\n", secs[u], n[u], 100 * hit[u] / n[u], u } ' access.log-2026090* | sort -nr | head -20 61402.8 s 67341 1.8% /reports/export 38915.3 s 204882 0.4% /catalog 22107.6 s 91120 11.2% /calendar/day 9044.1 s 1980433 96.7% /api/session The route with 1.9 million hits was the cheapest thing on the list. The one costing the most ran 67 thousand times at roughly 900 ms each, with a cache hit rate under 2%, because every request carried a different date range in the query string. Every filter combination that became a public address is a page that is generated once, served once, and cached for nobody. The two things that did not work robots.txt. Whoever respects it was already respecting it, and the traffic in that top bucket does not read it. Adding rules there changed nothing measurable in the following week. Blocking inside the application. The first attempt was a middleware that inspected the request and returned 403. In that stack the session middleware ran first, so by the time the rule said no, the request had already checked out a database connection. A 403 from the app measured around 34 ms and held a pool slot. The same 403 from nginx measured 0.4 ms. Saying no is not free, and where you say it is worth roughly two orders of magnitude. So the deny moved to the edge, per range rather than per address, since single IPs rotate too fast to matter. limit_req_zone $binary_remote_addr zone=perip:16m rate=3r/s; location /catalog { limit_req zone=perip burst=10 nodelay; limit_req_status 429; } Honest limit on that one: rate limiting by range punishes offices behind NAT, and I have no clean answer for telling those two apart at the edge. What actually moved the number The expensive route got a cheap variant for anything that had not requested an asset in the same connection window: same content, none of the aggregation queries that build the side panels. Query count per render went from 41 to 3. The cache key dropped the tracking parameters and the TTL went up on pages born from parameter combinations, which changed little between requests anyway. Filter combinations left the index, with canonical pointing at the unfiltered version. Two weeks later, summed upstream time was down about 44% with no new instance, and p95 improved on the routes humans actually use. This is not a niche problem, and it is worse on public code hosting While I was writing this up, Konstantin Ryabitsev, who runs kernel.org infrastructure, reported that git.kernel.org burns more CPU rendering commits as HTML for scrapers than it spends on every other kind of legitimate access combined, git clones included. Across five geo distributed nodes, fourteen cores are doing nothing else at any given moment. Simon Willison published the account and it landed on Hacker News. A git web frontend is the extreme version of the shape: commits times files times views, diff, blame, raw, tree, an address space that grows by multiplication. A person opens the same handful of those. A crawler opens all of them, once each, which defeats caching by design, because for every address the crawler request is both the first and the last. One thing the log does not answer. It will not tell you whether the collector eating your CPU feeds something that sends you users. Behaviour separates robots from humans, not wanted from unwanted, and that second call is not an infrastructure call. So, two questions for people who have been through this. How do you separate the crawlers you want from the ones you do not, when both arrive with a Chrome user agent from residential ranges? And has anyone found a rate limit granularity that survives corporate NAT without whitelisting by hand? Originally published on the Revin blog: https://revin.com.br/en/blog/scraper-traffic-cloud-bill