Here is a quiz that trips up experienced engineers. What is len("👩‍💻") in Python? The emoji shows up as one picture of a woman at a laptop, so you might say 1. Python says 3. Its UTF-8 encoding is 11 bytes long. And if you hand those 11 bytes to GPT-4's tokenizer, some of the resulting tokens are not characters at all, just fragments of one, bytes that mean nothing until their neighbors arrive.

That last fact sounds like a bug. It is the design. The tokenizers of GPT-2, GPT-4, and Llama 3 never see characters. They see bytes, and they learn which byte sequences to merge. That choice buys a guarantee no character-based tokenizer can offer, every possible string can be tokenized, with no unknown symbols, ever. To understand it, and the handful of hard bugs it creates, we need to be precise about what text actually is in a computer.

Characters, code points, and bytes

Unicode assigns every character a number called a code point, written U+ followed by hexadecimal digits. The letter A is U+0041, é is U+00E9, the euro sign € is U+20AC, the Chinese character 世 ("world") is U+4E16, and 😀 is U+1F600. Code points range from U+0000 to U+10FFFF, which is $17 \times 2^{16} = 1{,}114{,}112$ possible values, of which over 150,000 have been assigned to characters so far.

Two complications matter for tokenizers.

• What you see is not one code point. The é you type is usually the single code point U+00E9, but it can also be written as a plain e (U+0065) followed by a combining acute accent (U+0301). The two look identical and compare unequal. The woman-at-a-laptop emoji is three code points, 👩 (U+1F469), an invisible zero-width joiner (U+200D), and 💻 (U+1F4BB). What a reader perceives as one character is called a grapheme cluster, and it can span many code points.
• Code points are not bytes. Files, networks, and tokenizers deal in bytes, so code points must be encoded. The near-universal encoding is UTF-8, which uses one to four bytes per code point.

Python's str is a sequence of code points, so len counts code points. To see bytes, encode.

The two é's print the same and differ in every other column. Section 2.5 deals with that through Unicode normalization. Here we focus on the bytes.

UTF-8, bit by bit

UTF-8 is a beautifully engineered code, and every byte-level tokenizer depends on its properties, so let's derive it. A code point needs between 7 and 21 bits. UTF-8 packs those bits into one to four bytes, using the first few bits of each byte as a header.

The x's are the code point's bits, most significant first, left-padded with zeros to the payload width. Where do the range boundaries come from? A two-byte sequence carries $5 + 6 = 11$ payload bits, so it can represent values up to $2^{11} - 1 = \text{0x7FF}$. Three bytes carry $4 + 6 + 6 = 16$ bits, up to $\text{0xFFFF}$. Four bytes carry $3 + 6 + 6 + 6 = 21$ bits, which could reach $\text{0x1FFFFF}$, but Unicode stops at $\text{0x10FFFF}$, so four-byte sequences above that are forbidden.

Let's encode three characters by hand.

é, U+00E9. In binary, $\text{0xE9} = 1110\,1001$, eight bits. It is above 0x7F and at most 0x7FF, so it takes two bytes with 11 payload bits. Pad to 11 bits, 000 1110 1001, and split into 5 + 6, 00011 and 101001. Add the headers to get 110 00011 = 0xC3 and 10 101001 = 0xA9. So é is C3 A9.

€, U+20AC. Binary 0010 0000 1010 1100 is 16 bits, so three bytes with a 4 + 6 + 6 split, 0010, 000010, 101100. With headers, 1110 0010 = 0xE2, 10 000010 = 0x82, and 10 101100 = 0xAC. So € is E2 82 AC.

😀, U+1F600. That is 1 1111 0110 0000 0000, 17 bits, so four bytes with 21 payload bits. Pad to 0 0001 1111 0110 0000 0000 and split 3 + 6 + 6 + 6 into 000, 011111, 011000, 000000. With headers, 11110 000 = 0xF0, 10 011111 = 0x9F, 10 011000 = 0x98, 10 000000 = 0x80. So 😀 is F0 9F 98 80.

The cell checks all three against Python.

Inspect any text you like below. Each code point is shown with its UTF-8 bytes, header bits in one color and payload bits in another.

Why UTF-8 is designed this way

Each feature of the table has a reason, and each reason helps tokenizers.

• ASCII compatibility. Code points below 128 encode as themselves in one byte. Every ASCII file is already valid UTF-8, and English text costs one byte per character.
• Self-synchronization. Lead bytes (0, 110, 1110, 11110 prefixes) and continuation bytes (10 prefix) can never be confused. Dropped into the middle of a byte stream, you can find the next character boundary by skipping at most three continuation bytes. That is exactly what a streaming decoder needs (below).
• No false matches. Every byte of a multi-byte sequence is at least 0x80, so it can never be mistaken for an ASCII space, newline, slash, or quote. Splitting UTF-8 bytes on spaces never cuts a character in half.
• Sort order. Comparing UTF-8 strings byte by byte gives the same order as comparing their code points.

Invalid byte sequences

Not every byte string is valid UTF-8, and a decoder must reject (or replace) the invalid ones. There are five ways to go wrong.

1. A stray continuation byte (0x80 to 0xBF) where a lead byte was expected.
2. A truncated sequence, a lead byte promising more continuation bytes than follow.
3. An overlong encoding, a code point written with more bytes than needed. The slash U+002F could in principle be written as C0 AF, and early decoders that accepted this were tricked into letting "../" past security filters. The standard requires the shortest form, which is why the bytes 0xC0 and 0xC1 never appear in valid UTF-8 (any two-byte sequence starting with them would encode a value below 0x80), why 0xE0 must be followed by at least 0xA0, and why 0xF0 must be followed by at least 0x90.
4. A surrogate, U+D800 to U+DFFF. These 2,048 code points are reserved for the UTF-16 encoding and are not characters, so their UTF-8 forms (0xED followed by 0xA0 to 0xBF) are invalid.
5. A value above U+10FFFF, which is why 0xF4 must be followed by at most 0x8F and why the bytes 0xF5 to 0xFF never appear.

def utf8_encode(cp):
    raise NotImplementedError

def utf8_decode(data):
    raise NotImplementedError

Byte-level BPE

Now the tokenizer. The BPE of §2.2 started from characters. If the base vocabulary is characters, a tokenizer trained mostly on English has no entry for a character it never saw, say a rare Chinese character or a new emoji, and must map it to <unk>. Including every Unicode character as a base symbol would make the base vocabulary larger than the whole tokenizer.

Radford et al. (2019) made the switch that every GPT model since has kept. Run BPE on the UTF-8 bytes of the text. The base vocabulary is the 256 possible byte values, every string is a sequence of bytes, and so every string can be tokenized. As the GPT-2 paper puts it, a byte-level version of BPE needs a base vocabulary of only 256, instead of the more than 130,000 that a full set of Unicode symbols would need.

Merges then rebuild characters out of bytes where it pays off. Watch it happen on a tiny corpus mixing English, Hindi, Chinese, and emoji. We pre-tokenize with GPT-2's pattern (explained in the next part of this section), then run the training loop of §2.2 on byte strings.

The very first merges are not English at all. Every Devanagari character is three bytes that start with E0 A4 or E0 A5, so those byte pairs are the most frequent in the corpus, and BPE merges them before anything else. These first tokens are fragments, two bytes of a three-byte character. Two merges later the first full character, ी (bytes E0 A5 80), appears.

Real tokenizers are full of such fragments. Counting over their whole vocabularies, GPT-2 has 344 tokens whose bytes are not valid UTF-8 on their own, GPT-4's cl100k_base has 773, and GPT-4o's o200k_base has 1,562. In cl100k_base, the character 世 (bytes E4 B8 96) is split across two tokens, E4 B8 and 96. That has a practical consequence.

Decoding safely, one token at a time

When a model streams its answer, you decode and print tokens as they arrive. Decoding is concatenation of token bytes followed by UTF-8 decoding, and for a complete sequence that is all there is.

In plain words, glue the byte strings of the tokens together, then interpret the result as UTF-8. Decoding each token separately and joining the strings is not the same thing, because a token can end in the middle of a character. If you decode E4 B8 on its own you get garbage (Python's errors="replace" gives a replacement character �), and when 96 arrives you get another one, so the user sees "��" instead of 世.

The fix is a streaming decoder that holds back an incomplete character at the end of the buffer until the rest of it arrives. Thanks to self-synchronization, the incomplete part is easy to find. Look at the last three bytes for a lead byte whose sequence is not yet complete.

There is a second, rarer issue. Nothing forces a model to emit valid UTF-8. It can sample E4 followed by an ASCII letter, and the decoder must not crash. Production decoders replace invalid bytes with U+FFFD (tiktoken's decode uses errors="replace" by default), and ByT5's authors dropped illegal bytes from model output, though they report never observing their models produce any.

def stream_decode(pieces):
    raise NotImplementedError

In production you would use Python's incremental decoder, codecs.getincrementaldecoder("utf-8")(errors="replace"), which implements exactly this logic. Knowing why it exists is what the interview is about.

Making bytes printable, GPT-2's byte-to-unicode table

If you open GPT-2's vocabulary file you'll see tokens like Ġthe and Ċ. These are not strange characters in the vocabulary. They are a storage trick. GPT-2's reference code manipulated tokens as Python strings and saved the vocabulary as JSON, and raw bytes such as a space (0x20), a newline (0x0A), or 0x00 are awkward there (invisible, or split on, or not printable). So the code maps each of the 256 bytes to a printable Unicode character. The 188 bytes that are already printable characters (! to ~, ¡ to ¬, and ® to ÿ) map to themselves, and the other 68 bytes, in order, map to U+0100, U+0101, and so on. The space byte 0x20 is the 33rd of those, so it maps to U+0120, which is Ġ. Newline 0x0A maps to U+010A, which is Ċ.

The mapping is invertible and has no effect on what the model sees. It only affects how token strings are written down. Hugging Face's byte-level tokenizers use the same table, which is why you'll see Ġ in Llama 3 and Qwen vocabularies too.

def bytes_to_unicode():
    raise NotImplementedError

def visible(b):
    raise NotImplementedError

Pre-tokenization with regular expressions

Byte-level BPE on raw text has a problem that the GPT-2 paper describes well. BPE is greedy and frequency-driven, so it wastes vocabulary slots on many versions of common words, such as dog, dog., dog!, and dog?, and on junk that spans word boundaries, like the e␣ we found in §2.2. The fix is to split text into chunks before BPE, and only merge within a chunk. The GPT-2 tokenizer splits with a regular expression and, in the paper's words, prevents BPE from merging across character categories, with an exception for spaces.

Here is GPT-2's pattern. \p{L} means any Unicode letter and \p{N} any Unicode number.

's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+

The regex engine tries the alternatives left to right at each position and takes the first that matches. Reading them in order,

1. 's|'t|'re|'ve|'m|'ll|'d splits off English contractions, so "they're" becomes they + 're.
2. ?\p{L}+ is a run of letters with at most one leading space. This is the exception for spaces. The space before a word belongs to the word, so world and world are different tokens, and a sentence never needs separate space tokens between words.
3. ?\p{N}+ is a run of digits with an optional space. Letters and digits never share a chunk, so abc123 is two chunks.
4. ?[^\s\p{L}\p{N}]+ is a run of punctuation and symbols with an optional space. So dog. is always dog + ., and the dog problem disappears.
5. \s+(?!\S) is a run of whitespace, but it stops before the last whitespace character if a non-space follows. In "a␣␣␣b", it takes ␣␣ and leaves the last space to attach to ␣b through rule 2. Indented code keeps its word tokens like ␣return clean.
6. \s+ catches any whitespace left over, such as trailing spaces at the end of the text.

In plain words, a chunk is a word, a number, or a run of punctuation, each with at most one leading space, plus runs of whitespace, and BPE merges only inside a chunk. Python's built-in re module has no \p{L}, but [^\W\d_] (word characters that are not digits or underscores) matches the same letters for practical purposes, which is what our GPT2_RE above used.

GPT-4's tokenizer, cl100k_base, refined the pattern in several ways that are visible in the widget below.

'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+

• Case-insensitive contractions. (?i:...) matches 'S and 'LL too, so "HE'LL" splits like "he'll". GPT-2's pattern splits 'LL into ' and LL.
• Numbers in chunks of at most three digits. \p{N}{1,3} cuts "1234567" into 123, 456, 7, and numbers no longer take a leading space. This caps the number of number tokens at 1,110 (all strings of one to three digits) and makes their boundaries predictable. Section 2.6 shows what this does to arithmetic.
• Any single non-letter can lead a word. [^\r\n\p{L}\p{N}]?\p{L}+ lets a word absorb one leading character that is not a letter, digit, or newline, such as a space, a tab, an underscore, or a quote, so _bar and "Hello are chunks.
• Newlines are special. Punctuation can absorb trailing newlines ():\n is one chunk), and \s*[\r\n] groups whitespace that ends in a newline, which compresses code and prose with blank lines.
• Possessive quantifiers. The ?+ and ++ are possessive (they never give back what they matched), which keeps matching fast. Here they cannot change the chunks. After ?+ comes \p{L}+, which could never start on the non-letter that backtracking would hand back, and after ++ comes [\r\n]*, which always succeeds, so a backtracking engine would find the same matches more slowly.

GPT-4o's o200k_base goes further. It splits words at case changes (camelCase becomes camel + Case), attaches contractions to the word before them ( he'll is one chunk), and lets punctuation absorb a trailing slash. Compare all three on your own text.

Pre-tokenization buys more than clean merges. Because BPE never crosses a chunk boundary, a tokenizer can count distinct chunks once and train on that table (the deduplication trick from §2.2), and an encoder can cache chunk encodings. It also has costs. The rules are designed around English (only the ASCII apostrophe counts as a contraction, so "they’re" with a curly apostrophe splits differently), and languages written without spaces, like Chinese, Japanese, and Thai, turn whole phrases into one long chunk. And the rule against crossing spaces is a choice, not a law. Frequent multi-word expressions such as " of the" or " in order to" could be single tokens, and SuperBPE (Liu et al. 2025) learns ordinary merges first and then lifts the space restriction for a second stage of merges, which shortens English sequences substantially at the same vocabulary size.

import re

def pretokenize_gpt4(text):
    raise NotImplementedError

Putting it together, a complete byte-level tokenizer

You now have every piece of a GPT-style tokenizer. Pre-tokenize with a regex, turn each chunk into its UTF-8 bytes, train BPE on byte ids with the loop of §2.2, encode by merge rank inside each chunk, and decode by joining bytes before decoding UTF-8. Writing the whole thing, train, encode, and decode, with a round trip that holds for any string, is the first assignment of Stanford's CS336 and a standard interview question. In byte-level form the merge table is usually stored as a dict from a pair of ids to the new id, and new ids are handed out in order, 256, 257, and so on, so a merge's new id doubles as its rank.

def train_byte_bpe(text, num_merges, pattern):
    raise NotImplementedError

def encode(text, merges, pattern):
    raise NotImplementedError

def decode(ids, merges):
    raise NotImplementedError

Byte fallback, and what round trips promise

Not every tokenizer starts from bytes. SentencePiece (§2.4) builds its vocabulary from characters, which compresses better for scripts with many characters because a frequent Chinese character can be one base symbol instead of a three-byte merge chain. To keep the no-unknowns guarantee, it offers byte fallback. The vocabulary also contains 256 tokens named <0x00> to <0xFF>, and any character that is not in the vocabulary is written as the tokens of its UTF-8 bytes. Llama 1 and 2 used this, as Touvron et al. (2023) describe, falling back to bytes to decompose unknown UTF-8 characters. Gemma does too.

We can now say exactly what different tokenizers promise. The guarantee that matters, $\operatorname{decode}(\operatorname{encode}(s)) = s$, holds in three different senses.

The other direction, $\operatorname{encode}(\operatorname{decode}(\mathbf{w})) = \mathbf{w}$ for a sequence of token ids $\mathbf{w}$, fails for every tokenizer in this table. The ids for ␣ followed by world decode to " world", which re-encodes as the single token ␣world. Encoding always produces one canonical sequence, but many sequences decode to the same text. Models trained only on canonical sequences can behave strangely when fed non-canonical ones, a fact that §2.6 turns into a pitfall at prompt boundaries.

The real way, tiktoken

OpenAI's tiktoken is a fast, encode-only implementation of byte-level BPE (training happens elsewhere). Its API makes every concept in this section visible.

Notice _mergeable_ranks. tiktoken stores the tokenizer as a map from each token's bytes to its rank, rather than as a list of merge pairs. That is enough to encode, because the rank of a merged token equals the rank of the merge that created it. To encode a chunk, tiktoken repeatedly looks up the rank of the concatenation of each adjacent pair and merges the lowest, which is exactly the loop from §2.2.

In PyTorch, a byte-level model needs no tokenizer library at all. The ids are the bytes. Here is the input side of a ByT5-style model, with three special ids reserved below the 256 byte values (ByT5 reserves ids for padding, end of sequence, and unknown, and shifts the bytes up by three).

Read more

• Radford et al. (2019). Language Models are Unsupervised Multitask Learners. Section 2.2 introduces byte-level BPE and the rule against merging across character categories.
• OpenAI. tiktoken. The encoder behind GPT-3.5, GPT-4, and GPT-4o. tiktoken_ext/openai_public.py holds every pattern quoted in this section.
• OpenAI. GPT-2 encoder.py. The original regex, byte-to-unicode table, and BPE loop, in about a hundred lines.
• The Unicode Consortium. The Unicode Standard, Chapter 3, Conformance. The definitive rules for UTF-8, including the table of well-formed byte sequences (Table 3-7).
• Pike & Thompson (1993). Hello World, or Καλημέρα κόσμε, or こんにちは 世界. The paper by UTF-8's designers explaining why it works the way it does.
• Touvron et al. (2023). LLaMA, Open and Efficient Foundation Language Models. A short tokenizer paragraph with digit splitting and byte fallback.

Watch next

▶

▶