AI Engineer Exhaustive
HardPlusGLM-4.5

Transformer Block

GLM-4.5

Hard

Pre-norm GLM-4.5 block: RMSNorm, QK-norm attention with partial RoPE, residual, RMSNorm, dense-or-MoE FFN, residual.

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

A GLM-4.5 decoder layer is a pre-norm residual block: RMSNorm, attention, add; RMSNorm, feed-forward, add. The paper (Table 1 and §2.1) does not reprint this residual algebra. It specifies the pieces that sit inside it — grouped-query attention with partial RoPE, optional QK-Norm, and a feed-forward that is dense in the first few layers and a sigmoid-gated MoE afterwards.

That split is the reason the block is worth writing down as its own object. Early layers use a full-width SwiGLU MLP. Later layers replace that MLP with a sparse mixture (routed experts plus one shared expert). Attention is the same in every layer. GLM-4.5 turns QK-Norm on; GLM-4.5-Air leaves it off.

The official Glm4MoeDecoderLayer in Hugging Face Transformers implements exactly this sandwich. The zai-org/GLM-4.5 repository does not ship its own modeling file; the README points at Transformers, vLLM, and SGLang.

How it works

Let the residual stream be (x \in \mathbb{R}^{B \times T \times D}). For GLM-4.5, (D = 5120); for Air, (D = 4096). Both use (\varepsilon = 10^{-5}) RMSNorm.

\mathrm{RMSNorm}(u) = \frac{u}{\sqrt{\mathrm{mean}(u^2)+\varepsilon}} \odot \gamma.

The forward pass is pre-norm, not post-norm and not a sandwich:

x' = x + \mathrm{Attn}\!\big(\mathrm{RMSNorm}(x)\big),

y = x' + \mathrm{FFN}\!\big(\mathrm{RMSNorm}(x')\big).

(\mathrm{Attn}) is grouped-query attention. Queries, keys, and values are projected from the normalized stream. (Q,K) are RMSNorm-ed per head when use_qk_norm is true, then partial RoPE is applied to the first (\lfloor 0.5 \cdot d_{\mathrm{head}}\rfloor) channels of each head ((d_{\mathrm{head}}=128), so 64 rotary channels). Keys and values are repeated across query groups and mixed with the usual scaled dot-product. (Q,K,V) projections carry a bias; the output projection does not.

(\mathrm{FFN}) depends on the layer index (\ell). The config field first_k_dense_replace is the count of dense prefix layers (paper Table 1: 3 on GLM-4.5, 1 on Air):

\mathrm{FFN}_\ell(h) = \begin{cases} \mathrm{SwiGLU}_{\mathrm{dense}}(h) & \ell < k_{\mathrm{dense}}, \\ \mathrm{MoE}(h) & \ell \ge k_{\mathrm{dense}}. \end{cases}

Dense SwiGLU uses intermediate_size (12288 / 10944). The MoE path uses moe_intermediate_size (1536 / 1408) for each routed expert and for the shared expert, with top-8 routing over 160 (or 128) experts. Both SwiGLU flavors are the same three-projection form

\mathrm{SwiGLU}(h) = W_{\mathrm{down}}\big(\mathrm{SiLU}(W_{\mathrm{gate}} h) \odot W_{\mathrm{up}} h\big).

There is no extra residual scale, no attention output norm, and no layer-scale. Position embeddings are computed once at the model root and passed into every block as ((\cos,\sin)).

Official code

The layer is Glm4MoeDecoderLayer in modeling_glm4_moe.py (generated from modular_glm4_moe.py). The constructor picks Glm4MoeMLP vs Glm4MoeMoE with layer_idx >= config.first_k_dense_replace. The forward is input_layernorm → self_attn → residual → post_attention_layernorm → mlp → residual.

The same pre-norm stack appears in SGLang glm4_moe.py. The zai-org/GLM-4.5 README only points at those trees.

Watch-outs

Sources