AI Engineer Exhaustive
HardPlusGLM-4.5

MoE Forward

GLM-4.5

Hard

Sparse MoE forward combining group-routed top-k, SwiGLU experts, shared expert, and routed_scaling_factor.

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

Every GLM-4.5 layer after the dense prefix is a sparse mixture of SwiGLU experts plus one always-on shared expert. §2.1 and Table 1 fix the counts: 160 routed experts (128 on Air), 8 active per token, one shared expert, MoE intermediate width 1536 (1408 on Air). Routing is the sigmoid / loss-free / group-restricted gate of the companion note.

The paper does not write the mixer. Official Glm4MoeMoE does: dispatch each token to its top-(k) routed experts, weight those outputs by the (normalized, scaled) gate, then add the shared expert on the original residual. The shared expert is not gated and is not one of the (E) routed ids.

This is the compute path that makes the 355B model 32B-active (Air: 106B total, 12B active). Dense prefix layers use a full-width SwiGLU instead and never enter this module.

How it works

Input (h \in \mathbb{R}^{B \times T \times D}) is the post-attention, RMSNorm-ed stream. Flatten to (x \in \mathbb{R}^{N \times D}), (N=BT).

The router returns indices (\mathcal{T}(n)) of size (k=8) and weights (w_{n,i}) that already include optional probability normalization and the routed scale (\alpha):

w_{n,i} = \alpha \cdot \frac{s_{n,i}}{\sum_{j \in \mathcal{T}(n)} s_{n,j}+\varepsilon}, \qquad \alpha = \texttt{routed\_scaling\_factor}.

Each routed expert (E_i) is a bias-free SwiGLU with width (d_{\mathrm{moe}}):

E_i(x) = W_{\mathrm{down},i}\big(\mathrm{SiLU}(W_{\mathrm{gate},i} x) \odot W_{\mathrm{up},i} x\big).

(W_{\mathrm{gate},i},W_{\mathrm{up},i} \in \mathbb{R}^{d_{\mathrm{moe}} \times D}), (W_{\mathrm{down},i} \in \mathbb{R}^{D \times d_{\mathrm{moe}}}). Transformers stores the two up-projections packed as gate_up_proj[i] of shape ((2 d_{\mathrm{moe}}, D)) and splits them after the matmul.

The shared expert (E_{\mathrm{sh}}) is the same SwiGLU with intermediate size (d_{\mathrm{moe}} \cdot n_{\mathrm{shared}}). On both releases (n_{\mathrm{shared}}=1), so it is the same width as one routed expert, but it is a separate parameter set and runs on every token:

y_n = \sum_{i \in \mathcal{T}(n)} w_{n,i}\, E_i(x_n) + E_{\mathrm{sh}}(x_n).

Reshape (y) back to ((B,T,D)). That is the block’s FFN output; the decoder layer then adds the pre-FFN residual.

Two facts that are easy to miss:

  1. (\alpha) scales only the routed sum. The shared expert is added at scale (1). GLM-4.5 uses (\alpha=2.5); Air uses (\alpha=1.0).
  2. Dispatch is token-choice, not expert-choice. A token always activates exactly (k) routed experts plus the shared expert, regardless of how peaked (s) is.

Training-only extras from §2.4 (bias updates, sequence-level balance loss) do not change this inference sum.

Official code

Glm4MoeMoE, Glm4MoeExperts, and Glm4MoeTopkRouter in modeling_glm4_moe.py. The mixer is:

routed = experts(x, topk_indices, topk_weights)
return routed + shared_experts(residual)

with shared_experts = Glm4MoeMLP(..., intermediate_size=moe_intermediate_size * n_shared_experts).

SGLang’s Glm4MoeSparseMoeBlock in glm4_moe.py fuses the same sum; some kernels apply (\alpha) on the routed output instead of inside the router — never both. zai-org/GLM-4.5 points at those files and does not include its own MoE kernel.

Watch-outs

Sources