If you are reading your server logs to see how Googlebot crawls your site, there is one mistake that quietly ruins the entire analysis: trusting the Googlebot user agent string. Anyone can send a request that says Googlebot/2.1 in the user-agent header. Scrapers, bots, and competitors do it constantly to slip past crawler blocks. Across our own portfolio of sites we measured episodes where up to 56% of "Googlebot" hits came from IP addresses that were not Google at all. This guide shows you how to verify Googlebot the correct way.
Why the Googlebot user agent cannot be trusted
The user-agent header is just text the client sends. There is no signature, no secret, nothing to stop a script from claiming to be Googlebot. In one measurement window on a single site, spoofed Googlebot traffic swung between 0% and 56% of all requests carrying Google's user agent. That means if you count "Googlebot visits" by user agent, your crawl numbers can be off by a factor of two — and the error is episodic, so it creates false spikes and false drops that look like real crawl-behaviour changes.
The only reliable way: verify Googlebot by IP
Real Googlebot crawls from a published set of IP ranges. Google publishes these ranges as JSON, and verification is a two-step process:
- Check the IP against Google's official list. Google publishes
googlebot.json(crawlers) and related files atdevelopers.google.com/search/apis/ipranges/. If the client IP is inside those ranges, it is genuine Googlebot. - Or do a reverse-then-forward DNS lookup. Run a reverse DNS lookup on the IP — a genuine Googlebot IP resolves to a hostname ending in
googlebot.comorgoogle.com. Then run a forward lookup on that hostname and confirm it points back to the same IP. Both directions must match.
Most real Googlebot traffic comes from the 66.249. range, but never hardcode a single prefix — use the published list, because Google adds ranges over time.
A quick reverse-DNS check
On any Unix machine you can spot-check a suspicious IP from your logs:
host 66.249.66.1
# -> 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com
# -> must resolve back to 66.249.66.1If the reverse lookup does not end in googlebot.com/google.com, or the forward lookup does not match, it is not Googlebot — regardless of what the user agent says.
What about GoogleOther and other Google crawlers?
Google now runs several crawlers besides the indexing Googlebot — GoogleOther, Google-InspectionTool, and others — all from the same IP ranges but with different user agents and different purposes. GoogleOther is a general-purpose fetcher, not the indexer, so a site can see heavy GoogleOther activity while the indexing Googlebot has barely touched it. When you verify by IP you confirm it is Google; you still read the user agent to tell which Google crawler it was. The IP proves authenticity; the user agent tells you intent.
Why this matters for your SEO
Every conclusion you draw from crawl data depends on this. If you block "bad bots" by user agent you will eventually block real users or miss real scrapers. If you measure crawl budget by user agent, spoofing poisons the numbers. And if your site sits behind an edge cache (as it does on Cloudflare or any caching CDN), the picture is even more distorted, because Googlebot is served from the edge and never reaches your logs. See our companion piece on why Cloudflare hides most of Googlebot from your server logs.
At IT Reidarsson AB we verify every Googlebot claim against Google's published IP ranges as a matter of course, because our own data proved how badly user-agent counting misleads. If you want to know how search engines actually crawl and index your site — measured, not guessed — that is exactly what our website audits do.
