New pages sitting unseen? The question "how do I get Google to crawl my site" has a practical answer: you cannot command Googlebot, but you can remove every obstacle in its way and send it the right signals. Here is the checklist we run for our own sites and client audits, in order of impact.
1. Submit a sitemap to Google
A sitemap is the single most direct way to tell Google which URLs exist. Generate an XML sitemap listing your important pages, then submit the sitemap to Google in Google Search Console under Indexing → Sitemaps. One hard-won lesson from our own launches: if you use a sitemap index that points to child sitemaps, submit the child sitemaps individually too. We once had a site where only the index was submitted, and Google never stepped down into the children for three days — 1,791 pages stayed invisible until we submitted each child sitemap directly.
2. Request indexing for your most important pages
In Search Console, paste a URL into the URL Inspection tool and click Request Indexing. This pushes a single page into Google's crawl queue and often gets it fetched within hours. Two caveats from experience: the request quota is account-wide and resets daily, so spend it on your highest-value pages first; and conversion is variable — sometimes every requested page is crawled within hours, sometimes none are for days. Treat it as a nudge, not a guarantee.
3. Remove the things that block crawling
Google will not crawl what it cannot reach. Check each of these:
- robots.txt — make sure you are not disallowing the paths you want crawled.
- noindex tags — a stray
<meta name="robots" content="noindex">orX-Robots-Tagheader tells Google to drop the page. - Server errors — 5xx responses make Google back off crawling entirely.
- Orphan pages — a page with no internal links pointing to it is very hard for Google to discover. This is the most common cause of slow crawling on new sites.
4. Give Google links to follow
Googlebot discovers pages by following links. A brand-new page with zero internal links is at the back of the queue. Link to important new pages from your homepage, your navigation, and related articles. External links from other sites are even stronger — they are both a discovery path and an authority signal. See our guide on internal linking for SEO for how to structure this.
5. Use IndexNow for Bing (and the AI engines)
Google does not support the IndexNow protocol, but Bing does — and Bing feeds several AI assistants. Submitting your URLs via IndexNow opens a fast, independent channel that reaches Bing, ChatGPT and Copilot, separate from anything Google does.
How to check whether it worked
Do not trust the "indexed" count in the Sitemaps report — in our testing it is unreliable. Instead, use the URL Inspection tool for a definitive per-URL status, or search site:yourdomain.com/page. We cover this in detail in is my website indexed, and how to check.
Getting crawled is step one; getting indexed and then served in results are separate hurdles. If pages are crawled but not appearing, our website audit measures exactly where in that funnel your site is stuck.
