AI Engineer Exhaustive
HardPlusDeepSeek-V3

Transformer Block

DeepSeek-V3

Hard · 🔒 Plus required

MLA + MoE combined block

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

A DeepSeek-V3 block is a standard pre-norm Transformer residual pair whose two sublayers are not the usual multi-head attention plus dense MLP. Attention is always Multi-head Latent Attention (MLA). The feed-forward half is a dense SwiGLU MLP on a short prefix of layers and DeepSeekMoE on the rest. The paper keeps this skeleton so the model can cache a few hundred latent numbers per token instead of a full key/value tensor, while most of the parameter count lives in routed experts that a given token never touches. Production V3 has 61 such blocks; the first three use the dense MLP, the remaining 58 use MoE. The residual topology itself is ordinary: normalize, mix, add; normalize, mix, add.

How it works

Let (\mathbf{x}\in\mathbb{R}^{B\times T\times d}) be the residual stream at one layer, with (d=7168) in the 671B config. Both norms are RMSNorm with learnable gain and (\varepsilon=10^{-6}):

\mathbf{x}'=\mathbf{x}+\mathrm{MLA}(\mathrm{RMSNorm}(\mathbf{x})),\qquad \mathbf{x}''=\mathbf{x}'+\mathrm{FFN}(\mathrm{RMSNorm}(\mathbf{x}')).

MLA takes the normalized stream, compresses queries and the joint key/value state, applies decoupled RoPE on a 64-dimensional side channel, and writes an output of shape ((B,T,d)) through (W^{O}). The FFN branch is chosen from the layer index (\ell\in{0,\ldots,L-1}):

\mathrm{FFN}_{\ell}=\begin{cases} \mathrm{MLP}_{d\to d_{\mathrm{ff}}\to d} & \ell < N_{\mathrm{dense}},\\ \mathrm{MoE} & \ell \ge N_{\mathrm{dense}}. \end{cases}

The paper sets (L=61) and (N_{\mathrm{dense}}=3). Both MLP and every expert (shared or routed) are SwiGLU:

\mathrm{SwiGLU}(\mathbf{u})=W_{2}\bigl(\mathrm{SiLU}(W_{1}\mathbf{u})\odot W_{3}\mathbf{u}\bigr).

Dense layers use (d_{\mathrm{ff}}=18432). MoE layers use a much thinner inner width (d_{\mathrm{moe}}=2048), one always-on shared expert, and 256 routed experts of which (K_{r}=8) fire per token. The MoE residual in the paper already includes the incoming FFN input (\mathbf{u}_{t}) inside equation (12); the official Block instead adds that residual outside MoE.forward, so the module itself returns only shared-plus-routed expert outputs.

MLA and MoE do not share weights across layers. What they share is the residual width (d): MLA’s output projection and the FFN’s down-projection both land back in (\mathbb{R}^{d}), which is why the two halves compose without a reshape.

Official code

The block is Block in inference/model.py. Construction is:

self.attn = MLA(args)
self.ffn  = MLP(dim, inter_dim) if layer_id < n_dense_layers else MoE(args)
self.attn_norm = RMSNorm(dim)
self.ffn_norm  = RMSNorm(dim)

forward is two residual adds. Production numbers live in inference/configs/config_671B.json (n_layers=61, n_dense_layers=3, dim=7168). The ModelArgs defaults in the same file are a smaller demo (27 layers, one dense layer) used by the __main__ smoke test, not the 671B checkpoint.

Watch-outs

Sources