An over-broad Disallow can sweep up content I actually want indexed. The rule looks reasonable until I notice it also covers a folder of live pages. A Disallow covers URLs that should be indexed. Disallowed URLs are not crawled, so their content cannot be indexed or used by AI features.
Why does robots.txt blocks a directory with indexable content matter for AI search?
Disallowed URLs are not crawled, so their content cannot be indexed or used by AI features.
How do you fix it?
Narrow the Disallow to only non-indexable paths.
How do you find it on your site?
- List every Disallow rule and write down the paths each one covers.
- Cross-check those paths against your sitemap and key landing pages.
- Test a sample of important URLs in the Search Console robots.txt tester.
- Narrow any rule that is broader than it needs to be.
Which ranking and citation factors does it touch?
Indexable content behind a Disallow cannot accrue ranking or citation signals, because it is never crawled. This is lost opportunity rather than a penalty.
What is the impact?
High for the affected section, invisible to search and AI. Direct.
What is the evidence?
Disallowed pages cannot be crawled, indexed, or have their directives read. Google Search Central, Intro to robots.txt
Frequently asked questions about robots.txt blocks a directory with indexable content
What does "robots.txt blocks a directory with indexable content" mean?
A Disallow covers URLs that should be indexed.
How do you fix robots.txt blocks a directory with indexable content?
Narrow the Disallow to only non-indexable paths.