I see this most often when a staging Disallow: / survives a launch. It tells crawlers to stay out of the entire site, so pages can drop out of the index even though they load perfectly for humans. A blanket Disallow rule prevents crawling of the entire site.
Why does robots.txt disallow: / blocking whole site matter for AI search?
If Google cannot crawl, it cannot index or ground AI answers in your content; nothing else you do matters.
How do you fix it?
Remove or scope the Disallow. Verify in Search Console’s robots.txt report and URL Inspection.
How do you find it on your site?
- Open /robots.txt and look for User-agent: * followed by Disallow: /
- In Search Console URL Inspection, test a key URL and check whether it reports blocked by robots.txt.
- Use the robots.txt report in Search Console to test specific paths.
- If the site launched recently, compare the live robots.txt against the staging version.
Which ranking and citation factors does it touch?
Blocking crawl does not always remove a URL immediately, but it prevents recrawling and updates, and pages usually decay out of rankings and AI answers over time.
What is the impact?
Severe/blocking, total loss of organic and AI-feature visibility. Direct.
What is the evidence?
Crawlability is the precondition for indexing and for AI features, which use crawlable content. Google Search Central, Intro to robots.txt; Google Search Central, Optimizing your website for generative AI features on Google Search
Sources
Frequently asked questions about robots.txt disallow: / blocking whole site
What does "robots.txt Disallow: / blocking whole site" mean?
A blanket Disallow rule prevents crawling of the entire site.
How do you fix robots.txt Disallow: / blocking whole site?
Remove or scope the Disallow. Verify in Search Console’s robots.txt report and URL Inspection.