A URL blocked in robots.txt can still be indexed as a bare link with no description, because blocking crawl does not remove a URL from the index. To keep a page out, I allow crawling and use noindex, not a Disallow.
Why does indexed though blocked by robots matter for AI search?
Google can index a blocked URL it can’t read, producing a poor, contentless listing it can’t improve.
How do you fix it?
Allow crawling and add noindex if exclusion is wanted (the two are mutually exclusive).
How do you find it on your site?
- Search Google with the site: operator and look for URLs showing no available description.
- Check the Search Console Pages report for Indexed, though blocked by robots.txt.
- Decide whether each URL should be indexed or not.
- To remove it, allow crawling and add noindex, rather than relying on robots.txt.
Which ranking and citation factors does it touch?
A URL indexed without content adds nothing and clutters the index. The fix aligns crawl and index directives so the page is handled as intended.
What is the impact?
Medium-high; poor listings. Direct.
What is the evidence?
A robots-blocked URL can still be indexed; use noindex (crawlable) to exclude. Google Search Central, Block Search indexing with noindex; Google Search Central, Intro to robots.txt
Sources
Frequently asked questions about indexed though blocked by robots
What does "Indexed though blocked by robots" mean?
URL blocked in robots.txt yet still indexed (often URL-only).
How do you fix indexed though blocked by robots?
Allow crawling and add noindex if exclusion is wanted (the two are mutually exclusive).