Shared Expert

DeepSeek-V3

Easy

Always-active shared expert mechanism

Independent study note. Written from the public paper and official code. This is not TensorTonic Plus and does not reproduce their exercises, starter code, or tests. For the official version, subscribe on TensorTonic.

Overview

DeepSeekMoE isolates a small set of shared experts that see every token, and a much larger set of routed experts that see only the tokens the gate assigns. Shared experts are meant to absorb common, high-frequency computation so the routed specialists can differentiate. They are not gated: their output is added in full, with coefficient 1, for every position. On DeepSeek-V3 the paper uses N_s=1 shared expert per MoE layer and N_r=256 routed experts of intermediate width 2048. The first three layers stay dense; every later FFN is this shared-plus-routed mix.

How it works

Let \mathbf{u}_t be the (already RMS-normalized) FFN input. The paper writes the MoE residual block as

\mathbf{h}_t'=\mathbf{u}_t+\sum_{i=1}^{N_s}\operatorname{FFN}_i^{(s)}(\mathbf{u}_t)+\sum_{i=1}^{N_r}g_{i,t}\,\operatorname{FFN}_i^{(r)}(\mathbf{u}_t).

The shared sum has no g_{i,t}. Each \operatorname{FFN}^{(s)} is a SwiGLU of the same inner width as a routed expert (moe_inter_dim=2048 on 671B). Official inference fuses the $$ shared SwiGLUs into one MLP whose inner width is N_s\times moe_inter_dim. For the released 671B config $$, so fusion is a no-op and the shared path is a single SwiGLU

\operatorname{FFN}^{(s)}(\mathbf{u})=\mathrm{W}_2\bigl(\operatorname{SiLU}(\mathrm{W}_1\mathbf{u})\odot\mathrm{W}_3\mathbf{u}\bigr).

That tensor is computed for the full packed token batch and added to the scatter-sum of routed expert outputs. Shared parameters are not among the N_r centroids, do not receive a load-balance bias, and are not subject to node-limited top-k. During decode deployment the paper treats the shared expert as a permanently selected routed expert for placement (9 “experts” per token: 8 routed + 1 shared), but the mathematical coefficient remains 1, not a learned gate.

The residual \mathbf{u}_t+ in equation (12) is applied at Block level in the official graph (x = x + self.ffn(...)). MoE.forward itself returns shared + routed only.

Official code

inference/model.py, class MoE: self.shared_experts = MLP(args.dim, args.n_shared_experts * args.moe_inter_dim). Forward: z = self.shared_experts(x) on the flattened (T,d) batch, then (y + z).view(shape) after the routed loop. MLP is the same SwiGLU as a dense prefix layer (w1,w3 column-parallel, w2 row-parallel). 671B widths are in inference/configs/config_671B.json: n_shared_experts=1, moe_inter_dim=2048. Smaller ModelArgs defaults use n_shared_experts=2 and a fused inner width of 2\times 1408.

Watch-outs

• Multiplying the shared output by a router weight (or by 1/N_s) is not the paper's definition. Shared is always-on, unnormalized.
• Putting the shared expert into the top-k pool and hoping the gate always picks it is a different router. Official code runs shared outside Gate.
• Fusing N_s>1 into one wide SwiGLU matches the official inference file; implementing N_s separate modules is equivalent only if you sum them. Mixing both conventions double-counts.
• Shared experts still sit behind ffn_norm. Feeding raw block residuals into shared_experts skips the pre-norm the Block applies.

Sources

• Paper: DeepSeek-V3 Technical Report, arXiv:2412.19437 (§2.1 DeepSeekMoE, eq. 12; §4 N_s=1)
• Code: deepseek-ai/DeepSeek-V3 (inference/model.py, classes MoE and MLP)