RMSNorm

GLM-4.5

Easy

Root Mean Square Norm: stable, no-bias normalization used pre-attention and pre-FFN throughout GLM-4.5.

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

Root Mean Square Layer Normalization (Zhang & Sennrich, 2019) rescales a vector by its RMS and a learned gain. It does not subtract a mean and it has no additive bias. That is the default block norm in LLaMA-style decoders, and GLM-4.5 uses the same primitive everywhere a transformer block needs a pre-norm: before attention, before the FFN (dense or MoE), and once more on the stack output.

The GLM-4.5 paper barely derives the formula — it treats RMSNorm as known — but it does call the weights out in the optimizer section: Muon is used for almost every parameter except word embeddings, biases, and RMSNorm weights. That is a training detail, not a forward-pass change: at inference the module is still a gain-only RMS rescale. The released 355B and Air configs both set "rms_norm_eps": 1e-05.

Pre-norm (norm → sublayer → residual add) is what the official decoder implements. Post-norm would put the residual on the unnormalized stream and is not what these checkpoints expect.

How it works

For a hidden vector x \in \mathbb{R}^{D} with D = 5120 (355B) or $ = 4096$ (Air),

\mathrm{RMS}(x) = \sqrt{\frac{1}{D}\sum_{i=1}^{D} x_i^2 + \varepsilon}, \qquad \mathrm{RMSNorm}(x) = \gamma \odot \frac{x}{\mathrm{RMS}(x)}.

\gamma \in \mathbb{R}^{D} is learned, initialized to ones. \varepsilon = 10^{-5}. There is no \beta and no mean subtraction, unlike LayerNorm

\mathrm{LayerNorm}(x) = \gamma \odot \frac{x - \mu}{\sqrt{\sigma^2 + \varepsilon}} + \beta.

The official kernel upcasts x to fp32, computes the mean square and rsqrt, multiplies by \gamma, then casts back to the incoming dtype (bf16 in the public checkpoints). That extra precision on the reduction is what keeps the scale stable when $$ is thousands of channels.

Each decoder layer holds two of these at hidden width:

• input_layernorm — applied to the residual stream before GQA;
• post_attention_layernorm — applied to the post-attention residual before the dense SwiGLU or the MoE block.

The full model adds a third, model.norm, after the last layer and before the LM head. On GLM-4.5 only, two extra RMSNorms of width d_{\mathrm{head}} = 128 implement QK-Norm inside attention; those share the same class and \varepsilon, not the same \gamma.

Forward of a layer is therefore

h \leftarrow h + \mathrm{Attn}(\mathrm{RMSNorm}(h)), \qquad h \leftarrow h + \mathrm{FFN}(\mathrm{RMSNorm}(h)).

Shapes: residual h is (B, S, D). The reduction is always the last axis. Do not reduce over sequence or batch.

Official code

zai-org/GLM-4.5 ships no RMSNorm source. The README points at Transformers.

Glm4MoeRMSNorm in src/transformers/models/glm4_moe/modeling_glm4_moe.py is the reference. It is decorated with @use_kernel_forward_from_hub("RMSNorm") so a faster kernel can replace the Python path, but the math is the four-line fp32 mean / rsqrt / gain recipe above. The class docstring calls it equivalent to T5LayerNorm, which is the same gain-only RMS form.

Glm4MoeDecoderLayer wires input_layernorm and post_attention_layernorm. Glm4MoeModel owns the final norm. Config field: rms_norm_eps (default 1e-5) on Glm4MoeConfig.

Watch-outs

• Adding a bias, or subtracting a mean, is LayerNorm, not this checkpoint. The saved tensors are a single \gamma per norm.
• \varepsilon is 10^{-5}, not the 10^{-6} default in some LLaMA ports and not the constructor default 1e-6 on Glm4MoeRMSNorm when you instantiate the class without passing eps. Always pass config.rms_norm_eps.
• Compute the RMS in fp32. A bf16 mean of 5120 squares underflows the small terms and shifts every residual.
• QK-Norm uses this same class at D = 128. Reusing the block’s D = 5120 gain on Q or K is the wrong tensor.

Sources

• Paper: GLM-4.5 (arXiv:2508.06471), §2.1 (block layout) and §2.4 (Muon excludes RMSNorm weights)
• Code: zai-org/GLM-4.5; transformers Glm4MoeRMSNorm