AI Engineer Exhaustive
EasyPlusGLM-4.5

Partial RoPE

GLM-4.5

Easy

Apply RoPE only to the first rope_dim channels of each head; leave the remaining channels position-free.

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

Rotary Position Embedding (RoPE) encodes absolute position by rotating pairs of channels in the query and key. Full-head RoPE spends the entire head dimension on that rotation. Partial RoPE does not: it rotates only a prefix of each head and leaves the remaining channels as ordinary content features with no explicit position. GLM-4.5’s architecture section names this combination as “Grouped-Query Attention with partial RoPE.” The paper does not publish the split width. That number comes from the released checkpoint and the Hugging Face glm4_moe implementation that the official zai-org/GLM-4.5 README points to: partial_rotary_factor = 0.5 on a 128-wide head, so the first 64 channels rotate and the last 64 do not.

The design is a cheap way to keep some dimensions purely semantic. Those unrotated channels can still mix information across positions through attention, but their inner product does not change when both tokens shift by the same offset. That is useful when a wide head (128) would otherwise over-commit capacity to relative-position geometry. DeepSeek-V3 takes a related idea further with multi-latent attention and a dedicated RoPE subspace; GLM-4.5 stays with a simple channel split on an otherwise standard GQA head.

How it works

Each attention head has width d = 128. The rotary width is

d_{\mathrm{rope}} = \lfloor d \cdot \rho \rfloor, \qquad \rho = 0.5,

so d_{\mathrm{rope}} = 64. After the Q and K projections (and, on GLM-4.5 but not Air, after QK-Norm), split each head vector as

q = [q_{\mathrm{rot}};\; q_{\mathrm{pass}}], \qquad k = [k_{\mathrm{rot}};\; k_{\mathrm{pass}}],

with q_{\mathrm{rot}}, k_{\mathrm{rot}} \in \mathbb{R}^{d_{\mathrm{rope}}} and q_{\mathrm{pass}}, k_{\mathrm{pass}} \in \mathbb{R}^{d - d_{\mathrm{rope}}}.

Cosine and sine tables are built only for the rotary prefix. Their last dimension is d_{\mathrm{rope}}, not d. The official kernel uses LLaMA-style half-and-half pairing, not GPT-J interleaved even/odd pairs: if x = [x_1; x_2] with two blocks of length d_{\mathrm{rope}}/2,

\mathrm{rotate\_half}(x) = [-x_2;\; x_1].

The rotary update is then the usual complex multiply in that pairing:

q'_{\mathrm{rot}} = q_{\mathrm{rot}} \odot \cos m + \mathrm{rotate\_half}(q_{\mathrm{rot}}) \odot \sin m,

and the same for k at position n. Concatenate the untouched suffix:

q' = [q'_{\mathrm{rot}};\; q_{\mathrm{pass}}], \qquad k' = [k'_{\mathrm{rot}};\; k_{\mathrm{pass}}].

Values are never rotated. After this, GQA proceeds with the usual 1/\sqrt{d} scaled dot product over the full 128-wide heads. Shapes: $$ is (B, H_q, S, d) with H_q = 96; k is (B, H_{kv}, S, d) with H_{kv} = 8; \cos, \sin broadcast as (B, 1, S, d_{\mathrm{rope}}).

Relative-position identity still holds on the rotated slice: the inner product of two rotated prefixes depends on m-n. The pass-through slice contributes a position-invariant term q_{\mathrm{pass}}^\top k_{\mathrm{pass}}. The two terms add inside each head’s logit.

Official code

The zai-org/GLM-4.5 tree itself is inference wrappers and a README; it does not vendor a modeling file. Architecture lives in Hugging Face Transformers, which that README names as the reference implementation:

The modeling file is generated from modular_glm4_moe.py; edit that modular source, not the generated file, if you are patching Transformers.

Watch-outs

Sources