AI Engineer Exhaustive
EasyPlusGLM-4.5

Dense SwiGLU

GLM-4.5

Easy

SwiGLU FFN used in the dense prefix layer of GLM-4.5 (layers before the MoE blocks).

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

SwiGLU is the gated feed-forward block that replaced ReLU/GELU MLPs in most modern decoders (Shazeer, 2020): a SiLU-activated “gate” branch multiplied by a linear “up” branch, then projected back to the model width. GLM-4.5 uses that block in two places that share a class but not a width.

The first k layers after the embedding are dense — one SwiGLU, every token, no router. Table 1 calls these “# Dense Layers”: 3 on GLM-4.5 and 1 on Air. Their intermediate width is the “Dense Intermediate Dim,” 12288 (355B) or 10944 (Air). After that prefix, layers switch to MoE. The paper never writes “SwiGLU”; the activation is inferred from the official config "hidden_act": "silu" plus the three-projection MLP in Transformers. That is the standard SwiGLU parameterization, not a two-projection GeGLU or a squared-ReLU.

Those shallow dense layers are there because early residual stream features are shared by every token and are cheap relative to 160 experts. DeepSeek-V3 and Kimi K2 use the same “first-k dense, then MoE” pattern; GLM-4.5’s $$ matches DeepSeek-V3’s count while using a smaller dense width (12288 vs 18432) in keeping with the paper’s “narrower but deeper” design.

How it works

For hidden state x \in \mathbb{R}^{D} with D = 5120 and dense width d_{\mathrm{ff}} = 12288 (355B),

\mathrm{SwiGLU}(x) = W_{\mathrm{down}}\bigl(\mathrm{SiLU}(W_{\mathrm{gate}} x) \odot W_{\mathrm{up}} x\bigr),

\mathrm{SiLU}(z) = z \cdot \sigma(z) = \frac{z}{1+e^{-z}}.

All three maps are bias-free:

W_{\mathrm{gate}}, W_{\mathrm{up}} \in \mathbb{R}^{d_{\mathrm{ff}} \times D}, \qquad W_{\mathrm{down}} \in \mathbb{R}^{D \times d_{\mathrm{ff}}}.

The official Glm4MoeMLP.forward is exactly down(act(gate(x)) * up(x)). No extra scaling, no dropout inside the MLP (the paper sets dropout unused).

Layer index decides dense vs MoE. In Glm4MoeDecoderLayer:

\mathrm{mlp} = \begin{cases} \mathrm{Glm4MoeMLP}(d_{\mathrm{ff}}) & \ell < k, \\ \mathrm{Glm4MoeMoE} & \ell \ge k, \end{cases} \qquad k = \texttt{first\_k\_dense\_replace}.

For the 355B checkpoint k = 3, so layers 0,1,2 are dense SwiGLU and layers 3,\ldots,91 are MoE (92 transformer layers total, matching 3 dense + 89 MoE in Table 1; the extra MTP block is separate). Air uses $ = 1$ on 46 layers.

The surrounding block is pre-norm:

h \leftarrow h + \mathrm{Attn}(\mathrm{RMSNorm}(h)), \qquad h \leftarrow h + \mathrm{SwiGLU}(\mathrm{RMSNorm}(h)).

The residual add is outside the MLP. Do not fuse a second residual inside SwiGLU.

Shapes for a batch: x is (B, S, D); gate_proj/up_proj produce (B, S, d_{\mathrm{ff}}); the SiLU–mul stays in that shape; down_proj returns (B, S, D). Compared with one MoE expert (d_{\mathrm{moe}} = 1536), the dense prefix is eight times wider on 355B ($ = 8$). That is intentional: the prefix is a full-capacity FFN, not a single expert reused at the bottom of the stack.

The same Glm4MoeMLP class also implements the shared expert, but constructed with intermediate_size=moe_intermediate_size * n_shared_experts. This note is the dense-prefix path: omit that override so the module picks config.intermediate_size.

Official code

zai-org/GLM-4.5 does not include an FFN source file. Architecture is in Transformers:

vLLM and SGLang reimplement the same prefix rule; if you only have the Zhipu repo, start from its README pointer rather than guessing a path under inference/.

Watch-outs

Sources