RFC 9309 / START HERE
robots.txt crawl rules, before the details
REP means Robots Exclusion Protocol; robots.txt is the filename used to publish its rules.
A service publishes a plain-text file describing which URI paths automated clients called crawlers should visit or avoid. A crawler selects rules for its own name, then compares the requested path with the applicable Allow and Disallow patterns.
The idea in one picture
Fetch crawl rules
Match group + path
Apply crawl decision
A simplified picture of the idea; the chapters explain the actual mechanisms.
Why it exists
The protocol standardizes long-standing crawler conventions, including group selection, byte matching, errors and caching. RFC 9309 is a Proposed Standard from September 2022, with no formal updates or obsoletions in the checked metadata. These crawl rules are not access authorization and do not guarantee removal from a search index.
A familiar example
A fictional SampleBot fetches https://example.com/robots.txt. Disallow: /docs/ and a longer Allow: /docs/public/ make /docs/public/page.html allowed under REP. A 503 when fetching an uncached rule file instead means complete disallow, while a 404 permits access as an option. Real service authentication remains separate.
Helpful background
You can start here without reading another RFC. If these ideas are unfamiliar, these short introductions may help.
What you’ll learn next
Begin with chapter one, or choose a topic.
- Fetch and parse the service’s crawl rulesA crawler is an automated client, such as a search engine’s link follower. Before crawling a service, it retrieves robots.txt to find the owner’s requested rules for crawler access. These rules are not access authorization.→
- Select and combine crawler groupsThe crawler chooses a product token, a short name used to identify its relevant group. User-agent records name crawlers; the rest of the HTTP User-Agent header is not copied into that field.→
- Use the most specific matching pathAfter selecting the group, compare each Allow or Disallow pattern with the requested URI beginning at its first path byte. A longer matching rule can override a shorter one, regardless of file order.→
- Match wildcards and percent-encoded bytesPatterns can contain * for zero or more characters and $ to mark the end. Byte comparison also accounts for URI percent-encoding, so a literal special character is different from a wildcard operator.→
- Distinguish successful, unavailable and unreachableThe result of retrieving robots.txt changes what the crawler can assume. A file that the server says is unavailable is different from a file that cannot be reached because the server or network failed.→
- Follow redirects and bound cached rulesA robots.txt fetch may redirect, even to another host. The resulting rules still apply to the original authority. Caching can reduce refetches, but old rules normally should not be kept indefinitely.→
- Bound parsing without confusing rules with securityRule files are untrusted input, so a crawler needs safe parsing and matching limits. Separately, a Disallow record is a public request to cooperating crawlers, not a password wall or a promise of secrecy.→
This introduction explains RFC 9309 (2022). Read the original RFC ↗