Full Forward

GLM-4.5

Hard

Embedding plus stacked GLM-4.5 blocks (1 dense prefix layer, then MoE layers), final RMSNorm, lm_head logits.

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

The full GLM-4.5 forward pass is a decoder-only stack: token embedding, a shared RoPE cache, (L) pre-norm blocks, a final RMSNorm, and an untied linear lm_head. The paper’s architecture table (Table 1) fixes the widths and the dense/MoE split; it does not write the outer recurrence. That recurrence is what the official Glm4MoeModel / Glm4MoeForCausalLM actually run.

Two released sizes share the same graph and differ only in config. GLM-4.5 is 92 layers, hidden 5120, 3 dense prefix layers then 89 MoE layers. GLM-4.5-Air is 46 layers, hidden 4096, 1 dense prefix layer then 45 MoE layers. Teaching implementations often start from the Air split (one dense layer, then MoE). Both attach one extra MoE MTP module after the main stack; Transformers omits that module, while vLLM and SGLang load it for speculative decoding.

The stack is deep on purpose. §2.1 says the authors shrank width and expert count relative to DeepSeek-V3 / Kimi K2 and spent the budget on height, because deeper models scored better on reasoning.

How it works

Let token ids be (t \in {0,\dots,V-1}^{B \times T}) with (V = 151552). The embedding table (E \in \mathbb{R}^{V \times D}) produces the residual stream. RoPE frequencies are computed once from position_ids (base (\theta = 10^6) after the 32K mid-training extension; paper §2.4) and reused in every layer:

H^{(0)} = E[t], \qquad (\cos,\sin) = \mathrm{RoPE}(\mathrm{positions}).

Each layer (\ell = 0,\dots,L-1) is the pre-norm block of the companion note. The feed-forward inside that block is dense for (\ell < k_{\mathrm{dense}}) and MoE afterwards:

H^{(\ell)} = \mathrm{Block}_\ell\!\big(H^{(\ell-1)};\,\cos,\sin\big).

Released configs set (k_{\mathrm{dense}} =) first_k_dense_replace to 3 (355B) or 1 (Air). After the last block, a final RMSNorm sits outside the residual loop, then the language-model head:

\mathrm{logits} = H^{(L)}_{\mathrm{norm}}\, W_{\mathrm{lm}}^\top, \qquad H^{(L)}_{\mathrm{norm}} = \mathrm{RMSNorm}(H^{(L)}).

(W_{\mathrm{lm}} \in \mathbb{R}^{V \times D}) is not tied to (E) (tie_word_embeddings: false). Causal masking is the usual lower-triangular additive mask, including a padding mask when one is supplied.

Shapes that stay constant through the stack:

Attention uses GQA, partial RoPE ((1/2) of each head), and QK-Norm on the 355B model only. MoE layers use sigmoid group routing, one shared expert, and routed_scaling_factor (2.5) (355B) or (1.0) (Air).

The MTP module is not one of the (L) blocks. Checkpoints store it as model.layers.{L} (num_nextn_predict_layers = 1). The main causal-LM logits above do not pass through it.

Official code

Glm4MoeModel.forward and Glm4MoeForCausalLM.forward in modeling_glm4_moe.py are the reference: embed_tokens → rotary_emb → loop over self.layers → self.norm → lm_head. Config numbers live in the GLM-4.5 and GLM-4.5-Air config.json files.

zai-org/GLM-4.5 documents the series and points at Transformers / vLLM / SGLang. Transformers does not load the MTP layer; see glm4_moe_mtp.py in vLLM if you need that path.

Watch-outs

• The final RMSNorm is after the last residual add, not inside the last block. Skipping it, or applying a second norm before lm_head in the main path, mismatches the checkpoint.
• first_k_dense_replace is a count of dense layers at the start, not “replace the first (k) MoE layers.” Layer index (\ell \ge k) is MoE.
• RoPE is computed once and broadcast. Rebuilding (\cos,\sin) per layer is wasteful but correct; using a per-layer position_ids offset is not.
• Embeddings and lm_head are untied. Sharing them changes the output affine.

Sources

• Paper: arXiv:2508.06471, §2.1, Table 1, §2.4
• Code: huggingface/transformers glm4_moe; zai-org/GLM-4.5