Character Encoding, ASCII, and UTF

You save the name “Müller” in a file, but another application displays “Müller”. The applications are interpreting the same bytes using different rules.

Files and network messages store bytes. A character encoding specifies how to turn text into those bytes. Encoding writes the text as bytes; decoding reads the bytes back as text. The writer and reader need to use the same encoding.

ASCII stands for American Standard Code for Information Interchange. It defines 128 values for English letters, digits, punctuation, and control characters such as a line break. For example, A has the value 65, written as 41 in hexadecimal. Hexadecimal is base-16 notation, using 0–9 and A–F. ASCII uses seven bits, usually stored in one byte. It has no representation for ü, 日, or emoji.

Unicode assigns characters identifiers called code points. For example, ü has the code point U+00FC. The U+ prefix marks a Unicode code point, and 00FC is its number in hexadecimal. A code point identifies the character. An encoding determines which bytes represent it in a file.

UTF means Unicode Transformation Format. UTF-8 represents Unicode code points using one to four bytes. Its “8” refers to 8-bit units, rather than a fixed size for every character. Each line shows the character, its code point, and its UTF-8 bytes:

A   → U+0041  → 41             (1 byte)
ü   → U+00FC  → C3 BC          (2 bytes)
日  → U+65E5  → E6 97 A5       (3 bytes)
😀  → U+1F600 → F0 9F 98 80    (4 bytes)

Each pair, such as C3, represents one byte in hexadecimal. UTF-8 preserves ASCII values: A is stored as 41 in both encodings.

Step 1 — Assign a number. The Unicode Consortium assigned the character 😀, called GRINNING FACE, the number 128512. That assigned number is its Unicode code point. Step 2 — Write the number in hexadecimal. Decimal 128512 is hexadecimal 1F600. These are two ways of writing the same number. Step 3 — Add the Unicode notation. We write the code point as U+1F600. The U+ prefix tells the reader that this is a Unicode code point written in hexadecimal; the prefix adds no numeric value. Step 4 — Encode it for storage. UTF-8 encodes this code point into four bytes. We display those byte values in hexadecimal as F0 9F 98 80. U+1F600 is the character's identifier; F0 9F 98 80 is its UTF-8 byte sequence. Each byte is an 8-bit unit. Here are those same four bytes in binary:

F0 → 11110000 → 8 bits
9F → 10011111 → 8 bits
98 → 10011000 → 8 bits
80 → 10000000 → 8 bits

So UTF-8 stores 😀 using four bytes, or 32 bits. The 8 in UTF-8 refers to its 8-bit units. A character can require several of those units. Step 5 — Read and display it. A UTF-8 decoder reads the four bytes together and reconstructs code point U+1F600, or decimal 128512. The application then uses a font to display 😀.

In UTF-8, the name shown here has these bytes:

Müller → 4D C3 BC 6C 6C 65 72

A UTF-8 reader interprets C3 BC together as ü. A Windows-1252 reader interprets them separately as à and ¼, producing “Müller”. When importing a CSV file saved as UTF-8, choose UTF-8 in the import settings. If text is garbled, check the original file's encoding and reopen it using that encoding before saving changes.

UTF-8, UTF-16, and UTF-32 can represent the same Unicode text. They use different storage units: UTF-8: one to four bytes per encoded code point. UTF-16: one or two 16-bit units, or two or four bytes. UTF-32: one 32-bit unit, or four bytes. For new text files and web data, UTF-8 is a useful default. Use another encoding when the receiving system requires it. One visible character can also contain multiple code points, such as a letter followed by a combining accent. Counting visible characters, code points, and bytes can therefore give different results.