Fetch and parse the service’s crawl rules
01A crawler is an automated client, such as a search engine’s link follower. Before crawling a service, it retrieves robots.txt to find the owner’s requested rules for crawler access. These rules are not access authorization.
The malformed nonsense line is ignored while the later Disallow: /draft/ remains parseable and must be used. /robots.txt itself stays implicitly allowed.
Source for this drawing: RFC 9309 · Read RFC section 2.3 ↗
New to this? Start hereA quick picture + helpful background
A message needs an address.
Think of a packet as a small addressed envelope. Devices pass it onward; transport protocols decide what delivery promises an application gets.
Helpful first, if these ideas are unfamiliar
These pictures are analogies to help you learn. The linked RFC explains the actual rules.
Terms in this drawing
- Crawler
- An automated client that follows or fetches resources.
- Authority
- The service address portion of a URI, including host and any port.
Related RFCs and changes
“Updates” changes part of an earlier RFC. “Obsoletes” replaces it. This list shows selected related documents.
Rules MUST be accessible at the top-level, lowercase /robots.txt path. The file MUST be UTF-8 encoded and have media type text/plain. Successfully fetched parseable rules MUST be followed.
- Rules MUST be accessible at the top-level, lowercase /robots.txt path. The file MUST be UTF-8 encoded and have media type text/plain. Successfully fetched parseable rules MUST be followed.
- The discovery URI uses the service’s scheme and authority: scheme://authority/robots.txt. It is not /.well-known/robots.txt. The HTTPS example is a chosen service, not a rule that this protocol is HTTPS-only; the RFC also illustrates FTP.
- The format has groups of User-agent lines followed by Allow and Disallow rules. Field-name spelling is case insensitive under the ABNF; # introduces a comment, and optional whitespace is supported.
- Crawlers MUST try to parse each line and MUST use parseable rules. A malformed line does not justify ignoring a later valid rule. Rules before any group SHOULD be ignored.
- The /robots.txt URI is implicitly allowed. When no applicable rule matches a requested URI, access is allowed by this protocol; other authorization or service controls still apply.
All example.com URLs, SampleBot names and document paths are fictional. No crawler runs against those services. The picture separates fetching the rule file from evaluating subsequent crawl requests. The UTF-8/plain-text requirements apply to the file; this guide does not alter this site’s production robots.txt or sitemap.
Read the RFC section ↗