Here is a mistake that quietly invalidates a lot of crawl analysis: if your site sits behind an edge cache — as it does on Cloudflare, or any CDN with HTML caching — your server logs do not measure how Google crawls your site. They measure cache misses. We measured this on our own network, and the gap is enormous.
What we measured
On one site behind Cloudflare, we compared Googlebot activity at two layers on the same day: what the origin server logs recorded, versus what Cloudflare's edge analytics recorded. The origin logs showed 19 Googlebot requests. The Cloudflare edge showed 398. In other words, the origin server saw about 4.8% of the real Googlebot crawling — roughly 95% of it was served from Cloudflare's cache and never touched the origin at all.
Origin logs: 19 Googlebot hits. Edge: 398. The server saw under 5% of the truth.
Why this happens
When Cloudflare has your HTML cached at the edge, it answers Googlebot directly from the nearest data centre. The request is fully served without ever reaching your origin server — which means it never appears in your Apache or Nginx logs. Only cache misses reach the origin. So your server log is not a crawl log; it is a cache-miss log.
Why it matters
If you answer questions like "is Google crawling my new section?" or "did my crawl rate drop?" using origin server logs when an edge cache sits in front, your answers will be wrong — usually wildly understated, and distorted in ways that look like real changes. You will see phantom crawl drops that are really just cache warming, and miss real crawl activity entirely.
How to measure crawling correctly behind an edge cache
- Use Cloudflare's analytics, not your server logs. Cloudflare's GraphQL analytics API exposes requests at the edge, including cache status. Filtering for cache status other than "hit" gives you the discovery/crawl picture the origin cannot see.
- Use Search Console's Crawl Stats. Google reports its own crawl volume directly — it does not care about your cache.
- Verify Googlebot by IP, not user agent — spoofing distorts edge data too. See how to verify Googlebot.
The broader lesson
This is exactly the kind of measurement error that makes DIY SEO analysis unreliable — not because the data is wrong, but because it is the wrong data source for the question. Knowing where to measure is half the job. Our website audits pull crawl data from the layer that actually sees it, which is why our numbers match reality instead of the origin log's fiction.
