Week 6 — FP4 KV Caches and Bounded Replay
Quantization · KV Cache · Systems
Advanced
II — Memory-efficient inference
Overview
QAT is applied to the main global KV cache using an approximately four-bit E2M1 representation with E4M3 per-16-channel scaling, while SWA KV stays FP8 because it is more sensitive. SWA Bounded Replay reconstructs states from only the most recent n_win tokens.
What You Will Learn
- Explain quantization-aware cache design and per-channel scaling.
- Contrast persistent vs ephemeral state and exactness vs useful equivalence.
- Implement KV quantization at several bit widths and measure attention error.
- Implement exact vs bounded replay in a toy SWA network.
Core Concepts
Quantization error propagation
A single outlier channel can wreck a whole quantization group; per-channel/group scales are the fix.
Bounded replay
\text{exact replay}\sim L\,n_{\text{win}}\ \longrightarrow\ \text{bounded: replay only the last } n_{\text{win}}\ \text{tokens}
Why SWA stays FP8
Local sliding-window state is more quantization-sensitive than coarse global KV, so it is kept at higher precision while global KV is compressed to ~4 bits.
Prerequisites
Work these pages on this site before the lecture.
Lecture notes
Global KV is a good quantization target: it is large, persistent, and attended with a coarse index. The report’s recipe is roughly E2M1 values + E4M3 scale per 16 channels, trained with QAT. Local SWA KV stays higher precision (FP8) because a bad local scale wrecks the window.
Bounded replay reconstructs SWA state from only the last n_{win} tokens instead of persisting every layer’s window to SSD. Exact replay costs \sim L\,n_{win}; bounded replay pays a small quality tax for a large storage win.
Outlier channels still dominate group scales — the same lesson as LLM.int8().
Required readings
- DeepSeek-V4.1-Flash report — FP4 KV & bounded replay
- LLM.int8() (Dettmers et al.)
- Quantization chapter
Related on this site
Lab / Implementation
Mini-project: quantize synthetic KV tensors to 8/6/4/3/2-bit and measure attention-output error; then implement exact vs bounded replay in a toy multilayer SWA network.
Not on this site (paper reading required)
- FP4/E2M1/E4M3 specifics
- KV-specific QAT
- bounded replay (DeepSeek §2.5)
Mastery Check
- DERIVE: quantization error propagation and the replay error bound
- IMPLEMENT: KV quantizer + exact/bounded replay
- BENCHMARK: error vs bit width; quality vs compute saved
- DEBUG: an outlier channel destroying a group
- EXPLAIN: why SWA KV stays FP8 while global KV goes ~4-bit