Distinguish successful, unavailable and unreachable
05The result of retrieving robots.txt changes what the crawler can assume. A file that the server says is unavailable is different from a file that cannot be reached because the server or network failed.
Without a cached file, a server/network error leaves no available robots.txt rules. The crawler MUST assume complete disallow and stops before fetching content.
Source for this drawing: RFC 9309 · Read RFC section 2.3.1 ↗
New to this? Start hereA quick picture + helpful background
A message needs an address.
Think of a packet as a small addressed envelope. Devices pass it onward; transport protocols decide what delivery promises an application gets.
Helpful first, if these ideas are unfamiliar
These pictures are analogies to help you learn. The linked RFC explains the actual rules.
Terms in this drawing
- Unavailable
- The service reports that the rule file is unavailable, illustrated by 4xx.
- Unreachable
- The file cannot be reached due to server or network failure, illustrated by 5xx.
Related RFCs and changes
“Updates” changes part of an earlier RFC. “Obsoletes” replaces it. This list shows selected related documents.
On successful access, parseable rules MUST be followed. If the file is unreachable due to server or network errors, the crawler MUST assume complete disallow.
- On successful access, parseable rules MUST be followed. If the file is unreachable due to server or network errors, the crawler MUST assume complete disallow.
- A successful HTTP 200 in this illustration yields a parseable Disallow: /draft/ rule. A malformed unrelated line does not erase the valid rule.
- For an unavailable file, illustrated by HTTP 404, the crawler MAY access resources on the server. HTTP 400–499 is the RFC’s unavailable example range; this does not override any application authentication or a resource’s own HTTP response.
- HTTP 500–599 or network errors illustrate unreachable status. In the initial uncached example, complete disallow is required; do not treat an HTTP 503 as permission to crawl.
- If undefined for a reasonably long period, such as the RFC’s illustrative 30 days, crawlers MAY assume unavailable or continue using a cached copy. Thirty days is an example, not a fixed mandatory retry interval.
The three outcomes are separate branches, not one ordered exchange or a promise that a server transitions through all three. The unreachable illustration has no cached rule file; section 2.4 separately permits the unreachable cache exception. Rules concern crawl behavior and do not make protected resources readable.
Read the RFC section ↗