Inside the model · Tokenizer

Tokenizer

byte-level BPE · GPT-2 · real merges

base symbols
256 bytes
merges
50,000
vocab
50,257 256 + 50,000 + <|endoftext|>
this text
18 bytes → 5 tokens

All steps

  1. Pre-tokenize

    Why subwords: single characters make sequences very long, and whole words need a huge vocabulary that still misses new words; BPE sits in between. First a regular expression splits the text into words, numbers and punctuation, each keeping its leading space. Type your own text below.

    5 pieces

  2. Bytes

    Each piece becomes its UTF-8 bytes. GPT-2 gives every byte value a visible stand-in character, so a space is written Ġ.

    18 bytes

  3. BPE merges

    Inside each piece, the rank of every adjacent pair is looked up and the lowest-ranked pair is fused. This repeats until no pair is in the table; rare words end up as several tokens.

    13 merges fire

  4. Vocabulary ids

    Each final symbol is an entry in the vocabulary. A merged token’s id is 256 plus its merge rank. Ids 0–255 are the 256 byte symbols in GPT-2’s own order, not by byte value (“.” is 13, Ġ is 220), and 50256 is <|endoftext|>.

    ids [464, 3797, 3332, 319, 262]

  5. Decode

    Decoding runs the other way: each id is looked up to its byte string, the bytes are joined and read as UTF-8. A token can end in the middle of a character (try “café 你好”), so streaming output holds those bytes until the character is complete.

    ids → bytes → text

Code

import regex as re                  # pip install regex (\p{L} needs it)
pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""")

def encode(text):
    ids = []
    for piece in pat.findall(text):
        word = [byte_char[b] for b in piece.encode("utf-8")]    # " " → "Ġ"
        while len(word) > 1:
            pairs = list(zip(word, word[1:]))
            best = min(pairs, key=lambda p: ranks.get(p, float("inf")))
            if best not in ranks: break                         # no rule left
            word = merge(word, best)                            # every a, b → ab
        ids += [vocab[s] for s in word]                         # 256 + merge rank
    return ids

def decode(ids):
    data = b"".join(token_bytes[i] for i in ids)        # a lookup and a join
    return data.decode("utf-8", errors="replace")

Go deeper