AI Engineer Exhaustive
EasyPlusGLM-4.5

Shared Expert

GLM-4.5

Easy

Always-on expert that processes every token, summed with the routed mixture, with a routed scaling factor.

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

A Mixture-of-Experts layer usually routes each token to a sparse subset of feed-forward experts. That saves compute, but the token can miss a “default” transform that every position needs (a shared residual MLP). GLM-4.5 copies the DeepSeek-V3-style pattern of one shared expert plus many routed experts: Table 1 lists # Shared Experts = 1 for both 355B and Air, # Experts Active Per Token = 8, and 160 (355B) or 128 (Air) routed experts.

The paper states the counts and that they use loss-free balance routing with sigmoid gates. It does not write the shared-expert residual or the routed scale. Those are specified by the official Glm4MoeMoE module and the released configs: the shared expert is a dense SwiGLU whose intermediate width is moe_intermediate_size * n_shared_experts, its output is added to the routed mixture, and the routed mixture is multiplied by routed_scaling_factor after top-k weight normalization. That factor is 2.5 on GLM-4.5 and 1.0 on Air — a real architecture difference, not a leftover default.

How it works

Let x \in \mathbb{R}^{D} be one token (D = 5120 on 355B). The shared expert is an ordinary SwiGLU with intermediate size $ = 1536$:

E_{\mathrm{sh}}(x) = W_{\mathrm{down}}\bigl(\mathrm{SiLU}(W_{\mathrm{gate}} x) \odot W_{\mathrm{up}} x\bigr).

It runs on every token. No gate, no top-k.

In parallel, a sigmoid router scores the 160 routed experts, applies the loss-free correction bias, optionally masks by expert group, and keeps the top 8. If norm_topk_prob is true (it is on both public configs), those 8 sigmoid scores g_i are renormalized,

\bar{g}_i = \frac{g_i}{\sum_{j \in \mathrm{TopK}} g_j + \varepsilon}, \qquad \varepsilon = 10^{-20},

then scaled by \alpha = routed_scaling_factor:

w_i = \alpha\,\bar{g}_i.

Each chosen expert E_i is itself a SwiGLU of width d_{\mathrm{moe}}. The layer output is the sum, not a concat and not a residual around x inside this module (the decoder residual sits outside, around the whole MoE):

y = \sum_{i \in \mathrm{TopK}(x)} w_i\, E_i(x) \;+\; E_{\mathrm{sh}}(x).

Official code materializes this as routed = experts(x_flat, idx, w); y = routed + shared_experts(x_unflat). The shared branch reads the pre-routing hidden state (residuals = hidden_states), not the routed output. That matters: E_{\mathrm{sh}} is a function of x, not of the mixture.

Because n_{\mathrm{shared}} = 1, the shared MLP width equals one expert. If n_shared_experts were $$, the official constructor still builds one Glm4MoeMLP with intermediate size $ \cdot d_{\mathrm{moe}}$ — a single wider SwiGLU, not N separate modules. The public checkpoints use N = 1, so that distinction does not show up in the 355B/Air weights.

Shapes for a sequence: x is (B, S, D); the router flattens to (BS, D); routed experts write (BS, D) then reshape; E_{\mathrm{sh}} stays (B, S, D). The add is elementwise.

Dense prefix layers (the first first_k_dense_replace blocks) do not have a shared expert. They are a single wide SwiGLU. Shared experts exist only in Glm4MoeMoE, i.e. layers with index \ge 3 on 355B and \ge 1 on Air.

Official code

zai-org/GLM-4.5 README: implement via Transformers / vLLM / SGLang.

In src/transformers/models/glm4_moe/modeling_glm4_moe.py:

Watch-outs

Sources