AI Engineer Exhaustive
HardPlusDeepSeek-V3

Full Forward Pass

DeepSeek-V3

Hard · 🔒 Plus required

Complete DeepSeek V3 forward pass

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

The main-model forward pass of DeepSeek-V3 is a decoder-only stack: embed tokens, run (L) pre-norm blocks (MLA + dense-or-MoE FFN), apply a final RMSNorm, then a linear language-model head. That is the path that produces next-token logits. Multi-token prediction (MTP) is an extra training objective with its own sequential modules; it is described in Sec. 2.2 of the report and is not part of Transformer.forward in the official inference file. The published 671B model uses (L=61), hidden size 7168, vocabulary 129280, and activates about 37B of 671B parameters per token. Official inference is written for incremental decoding: it scores only the last position and keeps MLA caches across start_pos.

How it works

Let (t\in{0,\ldots,V-1}^{B\times S}) be token ids and (p) the cache offset (start_pos). The paper’s main model (ignoring MTP) is

\mathbf{h}^{(0)}=\mathrm{Emb}(t),\qquad \mathbf{h}^{(\ell+1)}=\mathrm{Block}_{\ell}(\mathbf{h}^{(\ell)}),\quad \ell=0,\ldots,L-1,

\mathbf{z}=\mathrm{Head}\bigl(\mathrm{RMSNorm}(\mathbf{h}^{(L)})\bigr)\in\mathbb{R}^{B\times S\times V}.

Each (\mathrm{Block}{\ell}) is the residual pair in the block note: MLA after attn_norm, then MLP if (\ell<N{\mathrm{dense}}) else MoE after ffn_norm. Production values: (L=61), (N_{\mathrm{dense}}=3), (d=7168), (n_{h}=128), (d_{c}=512), (d_{c}'=1536), (d_{h}^{R}=64), 256 routed experts / 8 active / 1 shared, sigmoid router.

The official inference forward is narrower than the training equation above. After the stack it slices the last time step before the head:

\mathbf{z}_{\text{infer}}=\mathrm{Head}\bigl(\mathrm{RMSNorm}(\mathbf{h}^{(L)})_{[:,-1,:]}\bigr)\in\mathbb{R}^{B\times V}.

RoPE frequencies are a prefix of a precomputed table: freqs_cis[p : p+S]. If (S>1), a causal mask (\mathrm{triu}(-\infty,1)) of shape ((S,S)) is added to attention scores; a single new token uses mask=None and relies on the KV cache length p+S. Under tensor parallelism the embedding table and the head are sharded; ParallelEmbedding all-reduces the summed partial embeddings, and the head all-gathers vocabulary shards.

MTP, when used at train time, sits beside this stack. Depth (k=1) reads (\mathbf{h}^{0}{i}) (the main-model hidden state) and (\mathrm{Emb}(t{i+1})), mixes them with (M_{1}\in\mathbb{R}^{d\times 2d}), and runs one extra Transformer block plus the shared output head. That path is specified in eqs. (21)–(25). It is not invoked by Transformer.forward.

Shapes for one 671B decode step ((B=1), (S=1)): residual ((1,1,7168)); cached per layer, (512+64=576) numbers; logits ((1,129280)). Prefill with (S>1) materializes a triangular mask and writes (S) cache slots starting at (p).

Official code

Transformer in inference/model.py:

  1. h = self.embed(tokens)
  2. freqs_cis = self.freqs_cis[start_pos:start_pos+seqlen]
  3. causal triu mask if seqlen > 1
  4. for layer in self.layers: h = layer(h, start_pos, freqs_cis, mask)
  5. h = self.norm(h)[:, -1]
  6. logits = self.head(h), then optional all_gather across the vocab split

Block / MLA / MoE / MLP implement the per-layer body. Hyperparameters for the real checkpoint are in inference/configs/config_671B.json. There is no MTP module in this inference file; the paper’s MTP section is the source for the training-time extra heads.

Watch-outs

Sources