Match wildcards and percent-encoded bytes
04Patterns can contain * for zero or more characters and $ to mark the end. Byte comparison also accounts for URI percent-encoding, so a literal special character is different from a wildcard operator.
%62 is the unreserved ASCII b, so /%62az compares as /baz. The reserved slash in /a%2Fb stays encoded. The non-ASCII é is represented by its UTF-8 percent-encoded bytes.
Source for this drawing: RFC 9309 · Read RFC section 2.2.3 ↗
New to this? Start hereA quick picture + helpful background
A message needs an address.
Think of a packet as a small addressed envelope. Devices pass it onward; transport protocols decide what delivery promises an application gets.
Helpful first, if these ideas are unfamiliar
These pictures are analogies to help you learn. The linked RFC explains the actual rules.
Terms in this drawing
- Wildcard
- The * operator matching zero or more characters.
- Percent-encoding
- Representing a byte as % followed by two hexadecimal digits.
Related RFCs and changes
“Updates” changes part of an earlier RFC. “Obsoletes” replaces it. This list shows selected related documents.
Crawlers MUST support # comments, * for zero or more characters and $ for the end of a match pattern. Literal special characters SHOULD be percent-encoded when matched verbatim in a URI.
- Crawlers MUST support # comments, * for zero or more characters and $ for the end of a match pattern. Literal special characters SHOULD be percent-encoded when matched verbatim in a URI.
- The chosen Disallow: /*.pdf$ matches /files/a.pdf. Its end anchor means /files/a.pdfx does not match that rule. All patterns used here start with /, avoiding the unverified leading-* grammar discrepancy.
- To match a literal asterisk in a filename, /file-%2A.txt uses an encoded byte rather than the * wildcard. Encoding a literal dollar similarly uses %24.
- Before comparison, non-ASCII octets and reserved octets require the percent-encoding treatment specified in section 2.2.2. A percent-encoded ASCII unreserved octet in the URI MUST be decoded; reserved or non-unreserved bytes remain encoded.
- Thus /%62az normalizes to /baz, while /a%2Fb does not become /a/b by decoding its reserved slash. The chosen character é is UTF-8 bytes C3 A9, represented as %C3%A9 for comparison. Do not count its Unicode code point as one octet.
These are worked matching examples, not a full URI-normalization implementation. A bare * means wildcard; %2A is a literal encoded byte. The figure-4 Unicode wording and leading-* ABNF discrepancy have reported errata 7128 and 7995; the guide uses independently derived UTF-8 values and slash-prefixed patterns instead of treating those reports as accepted changes. Query components require the RFC’s URI comparison treatment; a regex over decoded human-readable filenames alone is insufficient.
Read the RFC section ↗