Why Trend-Data Scrapers Silently Fail (And How I Fixed It)

You start a Google Trends scrape. It runs for twenty minutes. It comes back with nothing. You already paid for it. That's not a one-off bug, it's the most common complaint on scrapers in this category, across the board. Not "wrong data," not "missing feature." Just: ran, cost money, returned nothing. I wanted to understand why that keeps happening, and whether it's actually fixable. It's not Google Trends being flaky, it's IP reputation Google Trends doesn't have a public API. Every scraper, mine included, talks to the same internal endpoint the website itself uses. That endpoint is aggressively defensive: it rate-limits fast, and it treats requests from cloud/datacenter IPs very differently from requests that look like they come from an actual browser on an actual residential connection. I proved this to myself the boring way: I ran the exact same request from my own dev machine, no proxy, and got an instant 429. No warm-up, no browser fingerprint, just a blunt rejection. That's the failure mode hiding behind "ran 20 minutes, returned nothing." The scraper isn't broken, it's being told no, and most scrapers don't handle "no" gracefully, they just return nothing, and bill you anyway. The other bug: an anti-hijacking prefix that isn't consistent Separately, and this one genuinely surprised me, every JSON response from Google's Trends endpoint is prefixed with a classic XSSI guard to stop it from being naively eval'd if someone tricks a browser into loading it directly. Every writeup you'll find online shows this prefix followed by a comma. I hardcoded exactly that string, tested against a mocked server, shipped it, then ran it against the real endpoint. Parsing failed instantly. Turns out the separator after the guard isn't consistent, on the endpoint I hit, it's a bare newline, no comma at all. A one-character assumption, copied from a doc that was right for a different endpoint, and my "should work" client couldn't parse a single real response. The fix is boring on purpose: strip the guard sequence, then strip whatever whitespace or comma follows it, instead of matching one exact string. The lesson generalizes past this one field: testing against a mock proves your logic is internally consistent, not that it's correct. Only a live call against the real thing catches an assumption you didn't know you were making. What actually fixes the "paid for nothing" problem Two changes, neither exotic: Residential proxy, not datacenter. Once requests come from a residential-looking IP with a normal browser warm-up first, the blunt rejections mostly disappear. This is the single biggest lever, arguably bigger than anything in the code. Session rotation instead of blind retries. Retrying on the same blocked identity just burns time. My client opens a fresh proxy session, new IP, new cookies, after a failure, up to a capped number of attempts, before giving up on a request, with exponential backoff inside each session so retries don't hammer the endpoint in a tight loop, which is itself a great way to get blocked harder. And the part I think matters most: if every attempt still comes back empty, that gets logged clearly and the result simply isn't pushed to output. Not a silent empty row you already paid for, a clear line in the log saying it failed, and nothing charged for it. Built as an Apify Actor I packaged this as a Google Trends scraper on Apify, interest-over-time plus related queries, residential proxy by default, the retry and rotation logic above, and a hard rule that a keyword only shows up in the output, and only gets billed, if it actually came back with real data. If you've been burned by a trend-data scraper that quietly returned nothing, I'd genuinely like to know what you were pulling when it happened, I'm curious whether the pattern holds outside of what I tested. If you want to try it yourself: Reliable Trends Scraper on Apify