AI Engineer Exhaustive
EasyPlusArcee Trinity

Interleaved RoPE + NoPE

Arcee Trinity

Easy · 🔒 Plus required

3:1 RoPE/NoPE layer pattern

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

Every Trinity model repeats a four-layer attention motif: three local sliding-window layers that use RoPE, then one global causal layer that uses no positional encoding (NoPE). Section 2.2 calls this a 3:1 local:global ratio and attributes the layout to Yang et al. (2025). Trinity keeps QK-norm on both kinds of layer.

The split is an efficiency and long-context choice. Local layers cheapen attention at length and carry relative position; global NoPE layers let information travel the full prefix and, in their context-extension ablations, recovered loss faster when only the global layers were stretched.

How it works

Index layers from \ell=0. A layer is local if (\ell+1)\bmod 4\neq 0, global otherwise. On Large that is layer_types: three "sliding_attention" then one "full_attention", sixty times ($$ global layers). The same rule is the default in AfmoeConfig via global_attn_every_n_layers=4.

Queries and keys after QK-norm are

\widehat{\mathbf{q}}_{t,i}=\begin{cases}\mathrm{RoPE}(\mathbf{q}_{t,i}) & \text{local}\\ \mathbf{q}_{t,i} & \text{global (NoPE)}\end{cases} \qquad \widehat{\mathbf{k}}_{t,j}=\begin{cases}\mathrm{RoPE}(\mathbf{k}_{t,j}) & \text{local}\\ \mathbf{k}_{t,j} & \text{global}\end{cases}

with allowed sources (equation (8))

\mathcal{S}_{t}=\begin{cases}\{s\mid \max(1,t-w+1)\le s\le t\} & \text{local}\\ \{s\mid 1\le s\le t\} & \text{global.}\end{cases}

On Large, w=4096 and pretraining T=8192, so a local layer sees half the pretrain context; after extension the global layers grow (report: they extend global context and leave the local window fixed). RoPE uses \theta=10^{4} and rotates the full d_h=128 dims.

AfmoeModel still builds one (\cos,\sin) pair for the batch. AfmoeAttention applies apply_rotary_pos_emb only when is_local_attention. Global layers receive the same tuple and ignore it. Masks are chosen from a dict: sliding_attention vs full_attention.

The pattern is independent of the dense/MoE split. Layers 0,1,2 are local-dense; 3 is global-dense; 4,5 are local-dense; MoE starts at 6 (local). A 3:1 checker that also assumes “global iff MoE” is wrong.

Official code

Schedule construction is in configuration_afmoe.py (layer_types / global_attn_every_n_layers). Application is AfmoeAttention.forward (if self.is_local_attention) and the mask map in AfmoeModel.forward in modeling_afmoe.py. The paper statement is arcee-ai/trinity-large-tech-report §2.2.

Watch-outs

Sources