This is one of the few crawl issues I treat as an emergency. Google has said that if robots.txt returns a server error for long enough, it can stop crawling the whole site, so a flaky 500 on this single file can quietly suppress everything.
Why does robots.txt returns 5xx matter for AI search?
Google may treat a persistently unreachable (5xx) robots.txt as a signal to stop crawling the site entirely, because it cannot confirm what it is allowed to fetch.
How do you fix it?
Fix the server error so robots.txt returns 200 (with rules) or 404 (treated as allow-all). Monitor uptime of the file specifically.
How do you find it on your site?
- Run curl -I https://yourdomain.com/robots.txt and read the status code.
- Repeat at different times, because intermittent 5xx errors are the dangerous ones.
- Check the Search Console robots.txt report for fetch failures and the date they began.
- Check server and CDN logs for errors on the /robots.txt route specifically.
Which ranking and citation factors does it touch?
A persistent 5xx here can suppress crawling site-wide, which sits upstream of every ranking and citation signal. I fix this before anything else.
What is the impact?
High/blocking when it occurs, can suppress crawling site-wide until resolved. Direct, per Google’s documented handling.
What is the evidence?
Google’s documented handling: a server error for robots.txt can cause Google to pause crawling. Google Search Central, Intro to robots.txt; Google Search Central, How HTTP status codes affect Google Search
Sources
Frequently asked questions about robots.txt returns 5xx
What does "robots.txt returns 5xx" mean?
The robots.txt URL returns a server error (500-range) rather than 200 or 404.
How do you fix robots.txt returns 5xx?
Fix the server error so robots.txt returns 200 (with rules) or 404 (treated as allow-all). Monitor uptime of the file specifically.