Unicode

Overview

The Universal Character Set (UCS) is the coded character set used by Unicode. The UCS is divided in contiguous groups of $65, 536$ ( $2^{16}$ ) code points called planes. Plane $0$ is called the Basic Multilingual Plane (BMP). Planes $1$ through $16$ are called supplementary planes.

UCS-2

UCS-2 is an obsolete fixed-width encoding that assumed all representable characters could be identified by 2 bytes (hence its name).

UTF-16

UTF-16 is a variable-width encoding that superseded UCS-2.

Codes points "in the BMP" (i.e. less than $2^{16}$ ) are encoded using a single 16-bit code unit. Code points "above the BMP" (i.e. greater than or equal to $2^{16}$ ) are encoded using two 16-bit code units. These two code units are chosen from the surrogate range U+D800-U+DFFF. Values in this range are not used as characters, and UTF-16 provides no legal way to code them as individual code points.

Binary	Code Point	Range
`xxxxxxxxxxxxxxxx`	`xxxx xxxx xxxx xxxx`	`0x0000-0xFFFF`
`110110xxxxxxxxxx` `110111yyyyyyyyyy`	`xxxx xxxx xxyy yyyy yyyy +` `0x1000`	`0x10000-0x10FFFF`

If the encoding type is UTF-16BE, we assume a big-endian order. If UTF-16LE, we assume a little-endian order. If UTF-16, we rely on the byte order mark (BOM). This is a 16-bit code point with value U+FEFF that precedes the first actual coded value. If an opposite-endian decoder reads value U+FFFE, it knows the endianness must be opposite.

UTF-32

UTF-32 is a fixed-width encoding that stores each code unit in 4 bytes (32 bits). UTF-32 is effectively synonymous with UCS-32.

Like UTF-16, UTF-32 also has big- and little-endian variants.

UTF-8

UTF-8 is a variable-width encoding. Code points are stored using 1 to 4 bytes each. It uses a system of binary prefixes:

Binary	Code Point	Range
`0xxxxxxx`	`0xxx xxxx`	`0x0000-0x007F`
`110xxxxx 10yyyyyy`	`0xxx xxyy yyyy`	`0x0080-0x07FF`
`1110xxxx 10yyyyyy 10zzzzzz`	`xxxx yyyy yyzz zzzz`	`0x0800-0xFFFF`
`11110xxx 10yyyyyy 10zzzzzz 10wwwwww`	`000x xxyy yyyy zzzz zzww wwww`	`0x10000-0x10FFFF`

Without taking appropriate precautions, a particular code point may be encoded in multiple ways via overlong encodings. The following byte sequences are prohibited for this reason:

2-byte sequences: 0xC0 and 0xC1
3-byte sequences: 0xE0 followed by less than 0xA0
4-byte sequences: 0xF0 followed by less than 0x90

For compatibility reasons, UTF-8 prohibits 3-byte sequences corresponding to part of a UTF-16 surrogate pair (i.e. code points between U+D800 and U+DFFF). Lastly, no 4-byte sequence should decode to a value greater than U+10FFFF.

Normalization

A grapheme cluster is a string of one or more code points that make up a "user-perceived character". Every grapheme cluster is either composed, decomposed, or some combination of the two. This means two visually identical grapheme clusters may be comprised of different sequences of code points.

For example, the Vietnamese letter “ệ” can be expressed in five different ways:

Fully composed: U+1EC7 “ệ”
Partially composed: U+1EB9 “ẹ” + U+0302 “◌̂”
Partially composed: U+00EA “ê” + U+0323 “◌̣”
Fully decomposed: U+0065 “e” + U+0323 “◌̣” + U+0302 “◌̂”
Fully decomposed: U+0065 “e” + U+0302 “◌̂” + U+0323 “◌̣”

Visually identical grapheme clusters are said to be canonically equivalent. To handle canonically equivalent strings, normalization forms were introduced.