Crawl-delay set unreasonably high
Google ignores crawl-delay, but other crawlers including some AI bots honour it. A large value can slow those crawlers right down, and to me it usually signals a config copied without thought.
A large crawl-delay (which Google ignores, but other crawlers honour) signals confusion and slows non-Google bots.
Why it matters
Google does not support crawl-delay, but a high value indicates a possibly mismanaged file and can starve other engines.
How to fix it
Remove crawl-delay for Google; manage Google crawl rate via Search Console if needed.
How to find it on your site
- Search robots.txt for a Crawl-delay directive and note the value.
- Decide which non-Google crawlers you care about, since they are the ones affected.
- If the value is large, for example 10 or more, consider lowering or removing it.
- Manage Googlebot crawl rate through server capacity and Search Console, not crawl-delay.
Cross-reference to ranking and citation factors
Slower crawling of non-Google bots can delay how quickly AI engines pick up new or updated content, which affects freshness in those answers.
Impact
Low for Google specifically. Estimate.
Evidence
Google ignores crawl-delay and manages crawl rate itself. Google Search Central, Intro to robots.txt