AI Engineer Exhaustive
EasyPlusGPT-2

GELU Activation

GPT-2

Easy

Gaussian Error Linear Unit activation function

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

GELU (Gaussian Error Linear Unit) is the nonlinearity inside GPT-2’s position-wise feed-forward network. The GPT-2 paper does not write the formula: section 2.3 says the model “largely follows” the original GPT Transformer, and the released mlp uses GELU between the expand and project linears. That is the entire specification. Everything about the exact curve comes from Hendrycks & Gimpel (2016) plus the tanh approximation that OpenAI coded in src/model.py.

Unlike ReLU, GELU is smooth and is not a hard gate. It scales the input by the probability that a standard normal variate is smaller than that input. Negative values are attenuated rather than zeroed, which keeps a small gradient on the left tail. In GPT-2 this sits only in the MLP: attention is purely linear-plus-softmax.

How it works

The defining expression is

\mathrm{GELU}(x)=x\,\Phi(x)=x\int_{-\infty}^{x}\frac{1}{\sqrt{2\pi}}e^{-t^{2}/2}\,dt,

where (\Phi) is the standard-normal CDF. Equivalently, if (\varepsilon\sim\mathcal{N}(0,1)), then (\mathrm{GELU}(x)=x\cdot\mathbb{P}(\varepsilon\le x)). For large positive (x), (\Phi(x)\to 1) so GELU behaves like the identity. For large negative (x), (\Phi(x)\to 0) so the output vanishes. Near zero the function is slightly below the identity on the positive side and a shallow negative bowl on the left.

Evaluating (\Phi) with an erf is accurate but was not what the 2019 TensorFlow graph used. The official gelu is the widely circulated tanh approximation

\mathrm{GELU}(x)\;\approx\;\tfrac12 x\left(1+\tanh\left[\sqrt{\tfrac{2}{\pi}}\left(x+0.044715\,x^{3}\right)\right]\right).

In code that is a single expression: 0.5*x*(1+tf.tanh(np.sqrt(2/np.pi)*(x+0.044715*tf.pow(x, 3)))). The constant (0.044715) comes from a minimax-style fit to erf; it is not derived in the GPT-2 paper.

The MLP applies this elementwise after the first conv1d:

\mathrm{MLP}(x)=W_2\,\mathrm{GELU}(W_1 x+b_1)+b_2,

with (W_1) mapping (d\to 4d) (c_fc) and (W_2) mapping (4d\to d) (c_proj). For the default 124M-class hyperparameters, (d=768) so the hidden width is (3072). GELU is not used on the residual stream, on QKV, or on the tied LM head.

Two nearby approximations exist in the literature. BERT-style code often uses 0.5 * x * (1 + erf(x / sqrt(2))), which is the exact (\Phi) form. A later sigmoid form (\mathrm{GELU}(x)\approx x,\sigma(1.702x)) is cheaper and slightly less faithful. Matching a GPT-2 checkpoint requires the tanh polynomial above, not ReLU and not an arbitrary erf/sigmoid swap.

The function is applied independently at every batch, time, and channel index. There is no learned parameter inside gelu itself; all trainable scale lives in the surrounding conv1d weights and in the pre-norm affine terms.

Official code

See gelu and mlp in src/model.py of openai/gpt-2. mlp is called from block after ln_2. There is no separate activation module.

Watch-outs

Sources