AI Engineer Exhaustive
MediumPlusArcee Trinity

NoPE Layer

Arcee Trinity

Medium · 🔒 Plus required

Attention without positional encoding

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

A Trinity global layer is ordinary causal GQA with QK-norm and output gating, minus RoPE and minus the sliding window. Section 2.2 names this NoPE (Kazemnejad et al., 2023) and places one such layer after every three local RoPE+SWA layers. Position is not injected as an added vector either: there is no absolute PE table on these layers.

The paper’s reason is complementary roles. Local layers keep cheap relative geometry inside a window; global NoPE layers mix the full prefix. During context extension they found that stretching only the global layers recovered loss faster, which they read as evidence that the NoPE layers are the ones that must re-learn long-range structure.

How it works

After the attention pre-norm, the global layer still computes

\mathbf{q}^{0}_{t,i}=W^{Q}_{i}\mathbf{x}_{t},\quad \mathbf{k}^{0}_{t,j}=W^{K}_{j}\mathbf{x}_{t},\quad \mathbf{v}^{0}_{t,j}=W^{V}_{j}\mathbf{x}_{t}

and RMSNorms Q and K per head (equations (1)–(5)). The NoPE branch of (6)–(7) is the identity:

\widehat{\mathbf{q}}_{t,i}=\mathbf{q}_{t,i},\qquad\widehat{\mathbf{k}}_{t,j}=\mathbf{k}_{t,j}.

The source set is the full causal prefix, equation (8):

\mathcal{S}_{t}=\{s\mid 1\le s\le t\}.

GQA and gated SDPA are unchanged:

\alpha_{t,i,s}=\mathrm{Softmax}_{s\in\mathcal{S}_{t}}\Big(\frac{\widehat{\mathbf{q}}_{t,i}^{\top}\widehat{\mathbf{k}}_{s,j(i)}}{\sqrt{d_h}}\Big),\qquad \mathbf{o}^{\mathrm{sdpa}}_{t,i}=\sum_{s\in\mathcal{S}_{t}}\alpha_{t,i,s}\,\mathbf{v}_{s,j(i)},

then \widetilde{\mathbf{o}}_{t,i}=\mathbf{o}^{\mathrm{sdpa}}_{t,i}\odot\sigma(W^{G}\mathbf{x}_{t})_{i} and W^O. On Large, d_h=128, h_q=48, h_{kv}=8, so six query heads share each KV head. Softmax scaling is still 1/\sqrt{d_h}; NoPE does not remove that factor.

What a NoPE layer does still see: QK-norm (so logit scale stays controlled without RoPE), the sandwich norms around the sublayer, and the sigmoid output gate. What it does not see: \cos/\sin from AfmoeRotaryEmbedding, and sliding_window (set to None when layer_types[ℓ] != "sliding_attention").

Cache semantics: K,V stored for a global layer are unrotated. Replaying them later with RoPE, or mixing a local cache into a global layer, is invalid. The official module keys cache by layer_idx, so each layer keeps its own convention.

Official code

AfmoeAttention in modeling_afmoe.py sets is_local_attention from config.layer_types[layer_idx] and skips apply_rotary_pos_emb when false. The global mask is create_causal_mask in AfmoeModel.forward. There is no separate NoPE class. The paper definition is §2.2 of the tech report.

Watch-outs

Sources