RoPE Freqs

GLM-4.5

Easy

Base RoPE inverse-frequency table used by GLM-4.5 attention.

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

RoPE turns a position index m into a set of 2-D rotations, one rotation per frequency channel. The frequencies are not learned; they are a fixed inverse-frequency table

\theta_i = \mathrm{base}^{-2i / d_{\mathrm{rope}}}.

A larger base stretches the wavelengths, which is the standard knob for long context. GLM-4.5 pre-trains at 4,096 tokens with \mathrm{base}=10^{4}, then, when mid-training extends the sequence to 32K (and later 128K), “adjusts RoPE’s base frequency from 10,000 to 1,000,000” (§2.4). The public 128K checkpoints store that second value: "rope_theta": 1000000 and "rope_scaling": null. There is no YaRN, NTK-aware, or linear interpolation schedule on the released 355B/Air weights — just a larger \theta base and partial RoPE on half the head.

The paper specifies the two bases and the length schedule. It does not write the arange formula or the partial-width d_{\mathrm{rope}}. Those come from Glm4MoeRotaryEmbedding.compute_default_rope_parameters in Transformers, which the official GitHub README names as the model implementation.

How it works

Head width is d = 128. Partial RoPE uses \rho = 0.5, so the table is built for

d_{\mathrm{rope}} = \lfloor d \cdot \rho \rfloor = 64,

not for 128. There are d_{\mathrm{rope}}/2 = 32 frequencies. In the official construction (float arange, step 2):

\mathrm{inv\_freq}_j = \mathrm{base}^{-\,j / d_{\mathrm{rope}}}, \qquad j \in \{0, 2, 4, \ldots, d_{\mathrm{rope}}-2\}.

Equivalently, with i = 0,\ldots,31 and \mathrm{base} = 10^6 at inference,

\mathrm{inv\_freq}_i = 1000000^{-2i / 64} = 1000000^{-i / 32}.

Given integer positions p \in \mathbb{Z}^{B \times S}, form

\omega_{b,s,i} = p_{b,s} \cdot \mathrm{inv\_freq}_i

in fp32, then duplicate the vector so cosine/sine have length d_{\mathrm{rope}}:

\cos_{b,s} = \bigl[\cos\omega_{b,s};\; \cos\omega_{b,s}\bigr], \qquad \sin_{b,s} = \bigl[\sin\omega_{b,s};\; \sin\omega_{b,s}\bigr].

That duplication matches rotate_half: the first 32 rotary channels pair with the last 32. An attention-scaling multiplier exists on the module (attention_scaling) but is 1.0 for the default RoPE type, so it does not change the released model.

The same table is shared across all layers and all heads. It is computed once per forward from position_ids, not stored per layer. Cache-friendly decoding passes the running absolute index (past length plus the new step), not a restarted 0…S{-}1 range.

Why 10^6 instead of 10^4: under the original 10^4 base, the longest wavelength is on the order of 10^4 tokens. At 32K–128K those low-frequency channels wrap many times and relative-position signal at long range degrades. Raising the base stretches every wavelength by $$, which is the adjustment the paper reports when they leave the 4K pre-training regime.

Official code

zai-org/GLM-4.5 does not vendor the frequency kernel. See Transformers:

• Glm4MoeRotaryEmbedding.compute_default_rope_parameters in src/transformers/models/glm4_moe/modeling_glm4_moe.py — reads rope_theta (via config.rope_parameters), multiplies head_dim by partial_rotary_factor, then inv_freq = 1.0 / (base ** (arange(0, dim, 2) / dim)).
• forward — outer product of inv_freq with position_ids, cat((freqs, freqs), dim=-1), then cos / sin. Forced fp32 around that product.
• Released config.json: rope_theta = 1000000, partial_rotary_factor = 0.5, max_position_embeddings = 131072, rope_scaling = null.

If a path under the Zhipu repo looks like a custom RoPE module, treat it as unofficial; the README’s contract is “model code … in transformers, vLLM and SGLang.”

Watch-outs

• Building 64 frequencies over the full 128-wide head (or 32 frequencies over 128) desynchronizes cos/sin from apply_rotary_pos_emb, which sizes the rotary slice by cos.shape[-1].
• Using base = 10000 against the public 128K weights is the pre-training table, not the mid-trained one. Long-context behavior will be wrong even if the rest of the model matches.
• Integer division in arange(...) / dim (or a //) collapses many frequencies to zero. The official tensor is float32 on the exponent.
• Do not apply an extra YaRN/NTK scale unless you load a checkpoint that actually sets rope_scaling. The 355B/Air JSON leaves it null.

Sources

• Paper: GLM-4.5 (arXiv:2508.06471), §2.3–2.4 (4K → 32K → 128K; base 10^4 \to 10^6)
• Code: zai-org/GLM-4.5; transformers Glm4MoeRotaryEmbedding