Google only reads the first 500KB of robots.txt. Anything past that is ignored, so a bloated file can silently drop the rules I care about most if they sit near the end. The file exceeds Google’s size limit; content past the limit is ignored.
Why does robots.txt over 500kb matter for AI search?
Rules beyond the limit are not applied, so intended blocks/allows may silently fail.
How do you fix it?
Trim and simplify rules to stay under 500KB.
How do you find it on your site?
- Check the size with curl -s https://yourdomain.com/robots.txt | wc -c
- If it is near or over 500000 bytes, the tail is being ignored by Google.
- Audit for thousands of auto-generated Disallow lines, which is the usual cause.
- Consolidate rules with wildcards and move essential directives to the top.
Which ranking and citation factors does it touch?
Ignored rules can mean unwanted URLs get crawled or wanted ones stay blocked, both of which distort crawl efficiency rather than ranking directly.
What is the impact?
Medium. Mis-applied rules can cause unintended crawl behaviour. Direct (documented limit).
What is the evidence?
Google enforces a 500KB robots.txt limit; excess is ignored. Google Search Central, Intro to robots.txt
Frequently asked questions about robots.txt over 500kb
What does "robots.txt over 500KB" mean?
The file exceeds Google’s size limit; content past the limit is ignored.
How do you fix robots.txt over 500KB?
Trim and simplify rules to stay under 500KB.