robots.txt Disallow: / blocking whole site
I see this most often when a staging Disallow: / survives a launch. It tells crawlers to stay out of the entire site, so pages can drop out of the index even though they load perfectly for humans.
A blanket Disallow rule prevents crawling of the entire site.
Why it matters
If Google cannot crawl, it cannot index or ground AI answers in your content; nothing else you do matters.
How to fix it
Remove or scope the Disallow. Verify in Search Console’s robots.txt report and URL Inspection.
How to find it on your site
- Open /robots.txt and look for User-agent: * followed by Disallow: /
- In Search Console URL Inspection, test a key URL and check whether it reports blocked by robots.txt.
- Use the robots.txt report in Search Console to test specific paths.
- If the site launched recently, compare the live robots.txt against the staging version.
Cross-reference to ranking and citation factors
Blocking crawl does not always remove a URL immediately, but it prevents recrawling and updates, and pages usually decay out of rankings and AI answers over time.
Impact
Severe/blocking, total loss of organic and AI-feature visibility. Direct.
Evidence
Crawlability is the precondition for indexing and for AI features, which use crawlable content. Google Search Central, Intro to robots.txt; Google Search Central, Optimizing your website for generative AI features on Google Search
Sources