AI Engineer Exhaustive
MediumPlusGPT-2

Greedy Decoding

GPT-2

Medium

Autoregressive greedy generation

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

Greedy decoding is the deterministic way to turn GPT-2’s next-token distribution into a string: at every step, emit the single most probable token and feed it back as context. The paper uses this procedure in two zero-shot settings. Section 3.5 conditions on a document, dialogue history, and a final A: token, then greedily decodes answers on CoQA (55 F1). Section 3.7 seeds a few english = french pairs and greedily takes the first generated sentence as a translation.

The released sampler in src/sample.py is not greedy by default. sample_sequence divides logits by a temperature and draws from tf.multinomial. Greedy is the (T\to 0) / top_k=1 corner of that same loop. The paper also notes (section 3.6) that greedy summaries were more extractive and more repetitive than top-(k) samples, which is why summarization switched to (k=2).

How it works

A left-to-right language model defines

p_\theta(y_t\mid y_{<t},c)=\mathrm{softmax}\!\left(\ell_t\right)_{y_t},\qquad \ell_t=\mathrm{GPT2}([c;y_{<t}])[-1]\in\mathbb{R}^{V}.

Greedy generation is

\hat y_t=\arg\max_{v\in\mathcal{V}}\;p_\theta(v\mid \hat y_{<t},c) =\arg\max_{v}\;\ell_t[v].

No temperature, top-(k), or nucleus cutoff is applied: the argmax of the raw last-position logits is the next id. The new id is appended, and the model is run again until a length budget or a stop heuristic (first sentence, end-of-text, etc.).

In the official graph this is the same step used for sampling. model() returns logits of shape [B, T, V] and a per-layer KV cache present. A full-prefix first step encodes the prompt; later steps can pass only the newest token plus past, so attention keys grow on the sequence axis while queries stay length 1. positions_for must add past_length so the new token hits the correct wpe row.

Because (\arg\max) is invariant to any positive scale, dividing (\ell_t) by a temperature (T>0) does not change the greedy token. Temperature only matters once you sample. Setting top_k=1 in top_k_logits zeros every logit except the maximum (they are replaced by (-10^{10})), after which multinomial is forced to return that id. That is the practical greedy switch in this repo.

The paper’s translation and CoQA protocols are greedy plus a post-hoc cut: take tokens until a sentence boundary (translation) or treat the continuation after A: as the answer (CoQA). Those stops are evaluation wrappers, not part of sample_sequence. Unconditional scripts start from the <|endoftext|> id (generate_unconditional_samples.py).

Greedy is stable and cheap to evaluate, but it concentrates on high-mode continuations. The paper’s qualitative remark on summarization matches a known failure mode: short loops and copied spans once the model’s mode is “repeat the last clause.” Sampling methods trade that determinism for diversity.

Official code

The decode loop is src/sample.py: sample_sequence calls model.model each step, reads logits[:, -1, :], and (for greedy) you want argmax rather than tf.multinomial after temperature/top-(k). Drivers: src/interactive_conditional_samples.py and src/generate_unconditional_samples.py. The docstring on those scripts states that top_k=1 yields deterministic completions. Tokenization is src/encoder.py.

Watch-outs

Sources