AI Engineer Exhaustive
MediumPlusGPT-2

BPE Encode/Decode

GPT-2

Medium

BPE encoding and decoding

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

GPT-2 must map any Unicode string to a sequence of vocabulary IDs and invert that map without loss. Section 2.2 of the paper argues that a general language model should assign probability to every string, so the tokenizer cannot drop unknown characters or depend on a huge Unicode-code-point alphabet. The chosen compromise is byte-level Byte Pair Encoding (BPE), after Sennrich et al. (2015), run on UTF-8 bytes rather than on Unicode scalars.

Encoding is not “run BPE on the raw string.” The official Encoder first cuts text with a regex, maps each UTF-8 byte to a safe Unicode code point, then applies a ranked merge table, then looks up IDs. Decoding walks the same tables backward. The paper specifies the design (byte base, no UNK, reversible); the exact regex, the byte↔︎Unicode tables, and the greedy merge loop live in src/encoder.py.

This note is the runtime path. How the merge table is grown is a separate training problem.

How it works

The released files are encoder.json (string → integer, size 50,257) and vocab.bpe (ordered merge rules). get_encoder loads both and builds bpe_ranks: merge pair → rank (0 = first learned, highest priority).

Encode (Encoder.encode):

  1. Pre-tokenize. A regex (self.pat) splits text into pieces such as 's, optional-space + letters, optional-space + digits, leftover punctuation, and runs of whitespace. Contractions are split off so "don't" does not become one opaque blob.
  2. Bytes as symbols. Each piece is UTF-8 encoded. bytes_to_unicode maps every byte in (0\ldots 255) to a single Unicode character that is not whitespace or a control character the BPE loop would mishandle. Printable bytes keep a readable identity; the rest are remapped into a private range starting at 256.
  3. Greedy BPE. bpe treats the mapped string as a tuple of characters. While the current pair set is nonempty, it picks the pair with the smallest rank (unknown pairs have rank (\infty)) and splices that pair into one symbol. It stops when no remaining pair is in bpe_ranks, or the word is a single symbol. A cache memoizes the result per pre-token.
  4. IDs. The merged symbols, split on spaces that bpe inserts as delimiters, are looked up in encoder.

If (r(a,b)) is the rank of pair ((a,b)), each step replaces the currently cheapest adjacent pair:

(a,b)=\arg\min_{(x,y)\in\mathrm{pairs}(w)}\, r(x,y),\qquad r\notin\mathrm{ranks}\Rightarrow\infty.

Decode (Encoder.decode) is the inverse: IDs → merge-alphabet strings via decoder, concatenate, map each character through byte_decoder back to a byte, then UTF-8-decode the bytearray (errors='replace' by default).

The paper does not publish the regex or the bytes_to_unicode table. Those are official-code details. What the paper does fix is invertibility: because every byte has a symbol, every Unicode string has a token sequence, so a WebText LM can be scored on any benchmark regardless of that benchmark’s own tokenizer.

Official code

All of this is in src/encoder.py: bytes_to_unicode, get_pairs, Encoder.bpe / encode / decode, and get_encoder. Merge ranks come from vocab.bpe lines after the header; the token map is encoder.json under the model directory. The public repo does not re-train these files.

Watch-outs

Sources