AI Engineer Exhaustive
MediumPlusGLM-4.5

Group Router

GLM-4.5

Medium

Group-restricted top-k routing: sigmoid scores, pick topk_group groups first, then top-k experts within those.

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

GLM-4.5 routes each token to 8 of its routed experts with a sigmoid gate and the loss-free balancing recipe of Wang et al. (2024), cited in §2.1. The implementation, inherited from DeepSeek-V3, is group-restricted top-(k): experts are partitioned into (G) groups, the router first keeps topk_group groups, and only then picks num_experts_per_tok experts inside that subset.

The paper names the ingredients (sigmoid gates, loss-free balance, sequence-level auxiliary loss) but does not write the two-stage top-(k). That algorithm lives in Glm4MoeTopkRouter. Released GLM-4.5 and Air configs set n_group=1 and topk_group=1, so the group stage is a no-op on the published checkpoints — the same code path still has to implement it, because the kernel is shared with the DeepSeek-style layout and the config fields are part of Glm4MoeConfig.

Selection uses a learned per-expert correction bias. Mixture weights do not.

How it works

Flatten tokens so (x \in \mathbb{R}^{N \times D}). The gate is a bias-free linear map in float32, then a sigmoid:

s = \sigma(x W_g^\top) \in (0,1)^{N \times E}, \qquad W_g \in \mathbb{R}^{E \times D}.

(E) is 160 (GLM-4.5) or 128 (Air). Add the loss-free bias (b \in \mathbb{R}^{E}) only for choosing who fires:

s' = s + b.

Partition the (E) experts into (G=) n_group contiguous groups of size (E/G). A group’s score is the sum of its two largest bias-adjusted entries (DeepSeek’s noaux_tc rule, hard-coded as .topk(2) in Transformers):

g_g = \sum_{j \in \mathrm{top2}(s'_g)} s'_{g,j}, \qquad g = 1,\dots,G.

Keep the topk_group groups with the largest (g_g). Zero (or (-\infty)) the other groups. Among the survivors, take the top (k=8) experts by (s'):

\mathcal{T} = \mathrm{top}\text{-}k(s' \odot \mathbf{1}_{\text{kept groups}}).

Mixture weights are the raw sigmoid scores of those experts, not (s'). If norm_topk_prob is true (it is, on both releases), renormalize over the chosen set, then scale:

\bar{s}_i = \frac{s_i}{\sum_{j \in \mathcal{T}} s_j + \varepsilon}, \qquad w_i = \alpha\,\bar{s}_i, \quad i \in \mathcal{T}.

(\alpha) is routed_scaling_factor: (2.5) on GLM-4.5, (1.0) on Air. (\varepsilon = 10^{-20}) in the official kernel.

When (G=1) and topk_group=1, (g_1) is unused in practice and (\mathcal{T}) is ordinary top-8 over all experts. The general path is what the module implements.

The bias (b) is the loss-free balancer: it is updated from expert load during training (paper §2.4: update rate (0.001) for the first 15T tokens, then (0)) and stored as a buffer, not a gradient-trained weight. An extra sequence-level balance loss with weight (10^{-4}) is mentioned in the paper; it does not appear in the inference router.

Official code

Glm4MoeTopkRouter in modeling_glm4_moe.py. Current Transformers computes group_scores with .topk(2, dim=-1)[0].sum(-1), masks discarded groups, then topk for experts. SGLang wires the same fields into its fused MoE kernel (n_group, topk_group, scoring_func='sigmoid').

Config defaults are documented on the Transformers glm4_moe page. zai-org/GLM-4.5 does not ship a standalone router file.

Watch-outs

Sources