Bound parsing without confusing rules with security
07Rule files are untrusted input, so a crawler needs safe parsing and matching limits. Separately, a Disallow record is a public request to cooperating crawlers, not a password wall or a promise of secrecy.
For this illustrative protected resource, missing required credentials fail the service’s authentication check. REP’s path rules are not the gate enforcing that result.
Source for this drawing: RFC 9309 · Read RFC section 2.5 ↗
New to this? Start hereA quick picture + helpful background
A message needs an address.
Think of a packet as a small addressed envelope. Devices pass it onward; transport protocols decide what delivery promises an application gets.
Helpful first, if these ideas are unfamiliar
These pictures are analogies to help you learn. The linked RFC explains the actual rules.
Terms in this drawing
- Kibibyte
- 1024 bytes, abbreviated KiB.
- Access authorization
- The service’s decision about whether a requester may access content, separate from crawl preferences.
Related RFCs and changes
“Updates” changes part of an earlier RFC. “Obsoletes” replaces it. This list shows selected related documents.
Crawlers SHOULD impose a parsing limit, and that limit MUST be at least 500 kibibytes. REP is not a substitute for content security measures.
- Crawlers SHOULD impose a parsing limit, and that limit MUST be at least 500 kibibytes. REP is not a substitute for content security measures.
- One kibibyte is 1024 bytes. The minimum supported parsing-limit size is therefore 500 × 1024 = 512000 bytes, not 500000 bytes. This is a lower bound on a configured limit, not a required maximum file size.
- Implementors should handle memory and invalid characters safely and treat robots.txt as untrusted content. Apply the published grammar and preserve parseable rules within the supported processing scope.
- Listing Disallow: /private/ makes that path publicly discoverable. A conforming crawler follows the applicable restrictions, while another client can still send a request.
- A service that needs real access control must enforce an appropriate mechanism, such as HTTP Authentication. A request with no required credentials stops at that separate gate; robots.txt does not enforce it.
- Sitemaps and other extensions are outside the defined rule set here. They MAY be interpreted but MUST NOT disrupt core parsing. REP does not define an indexing-removal guarantee, a Crawl-delay directive, or testing authorization.
The size ruler is schematic; its lower-bound label is mathematical, not a simulated allocation of a huge file. No production crawling, indexing changes or access-control changes occur. A stopped authentication request represents a separate service requirement, not an automatic security feature conferred by a Disallow line.
Read the RFC section ↗