AI Engineer Exhaustive
EasyPlusDeepSeek-V3

Dense Prefix Layers

DeepSeek-V3

Easy · 🔒 Plus required

First N layers use dense FFN before MoE

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

DeepSeek-V3 is an MoE model, but not every layer routes. Section 4.2 of the report states that the authors “substitute all FFNs except for the first three layers with MoE layers.” Those first three blocks still run MLA for attention; only the feed-forward half stays a single dense SwiGLU. The official inference stack exposes the cutoff as n_dense_layers. Production 671B uses 3; the in-file ModelArgs demo uses 1. The paper does not prove why the prefix must be dense. The implementation fact is unambiguous: a contiguous prefix of layers is dense, and every later layer is MoE. Serving notes in Sec. 3 also treat “dense MLPs in shallow layers” as a distinct parallelism case (1-way TP, to avoid paying tensor-parallel communication on a relatively small MLP).

How it works

Let (\ell) be the 0-based layer index and (N_{\mathrm{dense}}) the prefix length. After the attention residual, the block applies

\mathbf{x}''=\mathbf{x}'+\mathrm{FFN}_{\ell}(\mathrm{RMSNorm}(\mathbf{x}')), \qquad \mathrm{FFN}_{\ell}=\begin{cases} \mathrm{MLP} & \ell < N_{\mathrm{dense}},\\ \mathrm{MoE} & \text{otherwise}. \end{cases}

For 671B, (N_{\mathrm{dense}}=3) and there are (L=61) layers, so layers (0,1,2) are dense and layers (3,\ldots,60) are MoE. The dense MLP is the same SwiGLU used inside each expert, but with a wide inner size:

\mathrm{MLP}(\mathbf{u})=W_{2}\bigl(\mathrm{SiLU}(W_{1}\mathbf{u})\odot W_{3}\mathbf{u}\bigr), \quad W_{1},W_{3}\in\mathbb{R}^{d_{\mathrm{ff}}\times d},\; W_{2}\in\mathbb{R}^{d\times d_{\mathrm{ff}}}.

config_671B.json sets (d=7168) and (d_{\mathrm{ff}}=\texttt{inter_dim}=18432). A later MoE expert uses (\texttt{moe_inter_dim}=2048) and, in V3, one shared expert plus 256 routed experts with top-8. The dense prefix therefore spends more FLOPs per token than a single expert, but far fewer than activating eight routed experts plus a shared expert, and it has no router.

Nothing in Sec. 2.1.2 (the MoE equations) applies inside the prefix: no sigmoid affinities, no bias load-balancing term, no node-limited routing. Attention is unchanged; MLA and its KV cache run on every layer, dense or sparse.

What is specified versus inferred: the count “first three layers” and the MoE replacement of all other FFNs are specified. The exact inter_dim=18432 is specified in the official 671B config, not in the prose of Sec. 4.2. The reason for a dense prefix is not derived in the V3 report; a common reading (early tokens need a shared, high-capacity mixer before expert specialization) is inference, not a paper claim.

Official code

Block.__init__ is the entire rule:

self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)

Transformer constructs Block(layer_id, args) for layer_id in range(n_layers). Configs:

MLP is column-parallel (W_{1},W_{3}) and row-parallel (W_{2}). Expert inside MoE is the same arithmetic without that parallelism.

Watch-outs

Sources