Google ignores crawl-delay, but other crawlers including some AI bots honour it. A large value can slow those crawlers right down, and to me it usually signals a config copied without thought. A large crawl-delay (which Google ignores, but other crawlers honour) signals confusion and slows non-Google bots.
Why does crawl-delay set unreasonably high matter for AI search?
Google does not support crawl-delay, but a high value indicates a possibly mismanaged file and can starve other engines.
How do you fix it?
Remove crawl-delay for Google; manage Google crawl rate via Search Console if needed.
How do you find it on your site?
- Search robots.txt for a Crawl-delay directive and note the value.
- Decide which non-Google crawlers you care about, since they are the ones affected.
- If the value is large, for example 10 or more, consider lowering or removing it.
- Manage Googlebot crawl rate through server capacity and Search Console, not crawl-delay.
Which ranking and citation factors does it touch?
Slower crawling of non-Google bots can delay how quickly AI engines pick up new or updated content, which affects freshness in those answers.
What is the impact?
Low for Google specifically. Estimate.
What is the evidence?
Google ignores crawl-delay and manages crawl rate itself. Google Search Central, Intro to robots.txt
Frequently asked questions about crawl-delay set unreasonably high
What does "Crawl-delay set unreasonably high" mean?
A large crawl-delay (which Google ignores, but other crawlers honour) signals confusion and slows non-Google bots.
How do you fix crawl-delay set unreasonably high?
Remove crawl-delay for Google; manage Google crawl rate via Search Console if needed.