RFC 2781 / START HERE
UTF-16, before the details
UTF-16 means the 16-bit Unicode Transformation Format. It represents Unicode numeric values as one or two 16-bit code units.
Unicode assigns numeric code points. A scalar is a code point outside the reserved surrogate range; a code unit is an integer used to represent it. UTF-16 uses one or two 16-bit units per scalar and writes each unit as two bytes in big-endian or little-endian order.
The idea in one picture
Choose scalar
Form code units
Order the bytes
A simplified picture of the idea; the chapters explain the actual mechanisms.
Why it exists
Software exchanges UTF-16 text in files and network formats. The recipient must know the byte order and handle surrogate pairs correctly. RFC2781 describes this representation and registers three charset labels; its Informational status does not make it an Internet Standard.
A familiar example
The same unit0x0102 (258 in decimal) can be bytes01 02 or02 01 (hexadecimal). A label or an appropriate initial byte-order mark distinguishes the chosen order. A valid decoding still does not prove that the text is safe to execute or appropriate for a media type.
Helpful background
You can start here without reading another RFC. If these ideas are unfamiliar, these short introductions may help.
What you’ll learn next
Begin with chapter one, or choose a topic.
- Encode one scalar into one or two code unitsA Unicode scalar is a numeric code point outside the reserved surrogate range. UTF-16 represents it with one or two 16-bit code units; bytes come later.→
- Decode a pair and stop at malformed inputA decoder distinguishes ordinary units, high surrogates and low surrogates. A high surrogate needs a following low surrogate before one scalar can be recovered.→
- Serialize each unit in the declared byte orderBig-endian writes a 16-bit unit’s higher-order byte first. Little-endian writes its lower-order byte first; it does not reverse the order of the code units.→
- Choose the sender’s charset labelThe MIME charset label tells a receiver how the octets are serialized. RFC2781 distinguishes known byte order, unknown byte order and formats that mandate a signature.→
- Interpret explicit BE and LE labels without switchingAn explicit UTF-16BE or UTF-16LE label fixes byte order. A conflicting initial marker is an error, rather than permission to silently swap the declared order.→
- Inspect the prefix of generic UTF-16The generic UTF-16 label permits either order. A conforming processor examines the first two octets, supports both orders and uses the RFC’s qualified fallback when no marker exists.→
- Keep a text FEFF separate from a signatureA byte-order mark is an initial signature in a chosen context. RFC2781 assigns a different meaning to FEFF inside text, and warns about concatenating independently signed strings.→
- Keep media rules and application safety separateEncoding labels describe how to recover character values. They do not override the surrounding media type or authorize executing decoded content.→
This introduction explains RFC 2781 (2000). Read the original RFC ↗