Week 3 — Causal Encoder–Decoder: Reducing Prefill

Architecture · Prefill · CED

Advanced

I — Long-context architecture

Overview

DeepSeek-V4.1-Flash splits a 40-layer language backbone into a 20-layer causal encoder and a 20-layer decoder. Decoder global KV is projected from the final encoder representation instead of every prompt token traversing the whole upper half; local SWA states stay layer-local.

What You Will Learn

• Derive the prefill complexity reduction.
• Explain why global KV can be generated differently from local KV.
• Build a two-stage transformer where the decoder K/V come from the encoder output.
• Reason about the information lost by the asymmetry.

Core Concepts

The complexity argument

O(NL)\rightarrow O\!\left(NL/2+n_{\text{win}}L/2\right)\approx O(NL/2)\quad(N\gg n_{\text{win}})

Asymmetric prefill/decode

Global KV is produced once at the encoder; each decoder layer keeps only its local SWA state. The design trades per-layer representation flexibility for prefill compute.

Prerequisites

Work these pages on this site before the lecture.

• Causal masking
• KV cache
• GPT pretraining

Lecture notes

CED splits the backbone: a causal encoder sees the prompt; a decoder consumes encoder-global KV plus its own sliding-window state. Prefill no longer walks every prompt token through every upper layer, which is the O(NL)\to O(NL/2) claim when N\gg n_{win}.

What you lose: each decoder layer does not get a unique global representation of the prompt. Debug for information leaks (decoder hidden states sneaking into global KV) before you celebrate the speedup.

YOCO and other cross-layer cache papers are the closest public cousins if you want a second derivation.

Required readings

• DeepSeek-V4.1-Flash report § CED / §2.2
• YOCO: You Only Cache Once
• KV cache chapter

Related on this site

• Causal masking · GPT pretraining
• Week 4 — CSA2

Lab / Implementation

Implement a toy two-stage transformer where the decoder obtains K/V projections from the encoder output; compare loss against a symmetric baseline at equal parameter count.

Not on this site (paper reading required)

• CED itself
• the complexity proof
• the SWA-state/global-KV distinction (DeepSeek §2.2)

Mastery Check

• DERIVE: O(NL/2) for N ≫ n_win
• IMPLEMENT: encoder-projected decoder KV
• BENCHMARK: loss vs symmetric baseline; prefill time vs N
• DEBUG: an information leak from decoder hidden states
• EXPLAIN: what is lost when global KV skips per-layer hidden states

Part of CS/AI 684 — Efficient Long-Context Agent Systems.