Select and combine crawler groups
02The crawler chooses a product token, a short name used to identify its relevant group. User-agent records name crawlers; the rest of the HTTP User-Agent header is not copied into that field.
OtherBot has no named group in this example, so it follows the wildcard group’s /archive/ restriction. A crawler with no match and no wildcard would have no applicable rules.
Source for this drawing: RFC 9309 · Read RFC section 2.2.1 ↗
New to this? Start hereA quick picture + helpful background
A message needs an address.
Think of a packet as a small addressed envelope. Devices pass it onward; transport protocols decide what delivery promises an application gets.
Helpful first, if these ideas are unfamiliar
These pictures are analogies to help you learn. The linked RFC explains the actual rules.
Terms in this drawing
- Product token
- The crawler name used to select rules, distinct from its full request header.
- Group
- One or more User-agent records followed by the rules that apply to those names.
Related RFCs and changes
“Updates” changes part of an earlier RFC. “Obsoletes” replaces it. This list shows selected related documents.
Crawlers MUST match product tokens case insensitively. When more than one group matches the user-agent, their rules MUST be combined before evaluating paths.
- Crawlers MUST match product tokens case insensitively. When more than one group matches the user-agent, their rules MUST be combined before evaluating paths.
- A crawler product token MUST contain only letters, underscores and hyphens. SampleBot is valid; a version string such as SampleBot/1.0 is not the product-token syntax. The token SHOULD be a substring of the identification string sent to the service.
- Multiple consecutive User-agent lines can share one rule group. Another User-agent line after the rules starts a new group. Blank lines do not by themselves terminate a group.
- If no matching group exists, the crawler MUST obey a User-agent: * group if present. A named matching group is not automatically combined with that fallback group.
- If no named group and no * group match, no rules apply. An empty applicable group likewise has no restrictions. Other records, such as a Sitemap extension, MUST NOT interfere with defined records or terminate a group.
The drawings use exact case-insensitive token matches, not a search-engine-specific substring ranking algorithm. Two SampleBot groups are merged; OtherBot uses the wildcard fallback. The RFC does not define a comma-separated user-agent extension here, and reported erratum 8895 is not adopted as permission to introduce one.
Read the RFC section ↗