Follow redirects and bound cached rules
06A robots.txt fetch may redirect, even to another host. The resulting rules still apply to the original authority. Caching can reduce refetches, but old rules normally should not be kept indefinitely.
At 25 hours old, the example’s current fetch fails with HTTP 503. The unreachable exception permits continued cached use; it is not an unconditional fresh-cache claim.
Source for this drawing: RFC 9309 · Read RFC section 2.3.1.2 ↗
New to this? Start hereA quick picture + helpful background
A message needs an address.
Think of a packet as a small addressed envelope. Devices pass it onward; transport protocols decide what delivery promises an application gets.
Helpful first, if these ideas are unfamiliar
These pictures are analogies to help you learn. The linked RFC explains the actual rules.
Terms in this drawing
- Redirect chain
- Consecutive responses that point retrieval to another URI.
- Cache age
- Elapsed time since this stored rule file was fetched.
Related RFCs and changes
“Updates” changes part of an earlier RFC. “Obsoletes” replaces it. This list shows selected related documents.
Crawlers SHOULD follow at least five consecutive redirects, even across authorities. If a file is reached within five, it MUST be fetched, parsed and followed in the initial authority’s context.
- Crawlers SHOULD follow at least five consecutive redirects, even across authorities. If a file is reached within five, it MUST be fetched, parsed and followed in the initial authority’s context.
- The worked example reaches a file at rules.example.com after exactly five redirects from example.com. Its rules govern the original example.com context, not automatically every resource at the final host.
- After more than five consecutive redirects, crawlers MAY assume the file is unavailable. At least five is not an unconditional five-redirect maximum.
- Crawlers MAY cache robots.txt and MAY use standard HTTP cache control. They SHOULD NOT use a cached version for more than 24 hours unless robots.txt is unreachable.
- The worked cache was fetched at 2026-10-20T08:00:00Z. At next-day 07:00 it is 23 hours old. At next-day 09:00 it is 25 hours old; continued cache use is illustrated only with the unreachable exception, not as a new general freshness rule.
Redirect arrows are a compressed schematic of five HTTP redirect hops, not a claim that there are only two hosts in every chain. Cache age is computed from fixed fixture timestamps, independent of the browser clock. The 24-hour statement is SHOULD NOT with an explicit exception, not a mandatory expiry at exactly 24 hours.
Read the RFC section ↗