Week 4 — CSA2: Sparse Attention Across Layers

Sparse Attention · KV Reuse · Systems

Advanced

I — Long-context architecture

Overview

The central week. CSA2 combines sequence compression with cross-layer KV reuse and sparse-index reuse. Three modes — Full, Reindex, Reuse — all compute their own main query and SWA KV but differ in whether main KV, indexer K, and Top-K indices are recomputed or reused.

What You Will Learn

• Reconstruct the three modes tensor-by-tensor from memory.
• Explain cross-layer KV sharing vs blindly sharing routing.
• Build a 6-layer stack with a mixed mode schedule.
• Measure the quality/cache tradeoff as mode frequency changes.

Core Concepts

Full

Q_l,\ KV_l,\ K_l^{idx}\rightarrow \text{score}\rightarrow \text{Top-K}\rightarrow \text{attention}

Reindex

Q_l,\ KV_{\text{shared}},\ K_{\text{shared}}^{idx}\rightarrow \text{new scoring}\rightarrow \text{new Top-K}

Reuse

Q_l,\ KV_{\text{shared}},\ \text{Top-K}_{\text{shared}}\rightarrow \text{attention}

Prerequisites

Work these pages on this site before the lecture.

• Causal masking
• Dot product
• KV cache

Lecture notes

CSA2 is the course center. Every layer still computes its own query and its local SWA KV. The three modes differ only in the global / indexer tensors:

• Full — compute main KV, indexer K, scores, and Top-K.
• Reindex — reuse shared main KV and indexer K; recompute scores and Top-K.
• Reuse — reuse shared KV and the previous Top-K; only attend.

Reuse is not “share routing everywhere.” A Reuse layer that reads stale indices after the prompt grew is the canonical bug. Your 6-layer lab schedule Full → Reuse → Reuse → Reindex → Reuse → Reuse is chosen so you can see both reuse and a mid-stack refresh.

Required readings

• DeepSeek-V4.1-Flash report — CSA2
• DeepSeek-V3 (MLA + sparse MoE context)
• YOCO

Related on this site

• Assignment A2
• DeepSeek sparse MoE

Lab / Implementation

Build the 6-layer stack Full → Reuse → Reuse → Reindex → Reuse → Reuse and log cache bytes and compute per layer; sweep Full/Reindex frequency (Assignment 2).

Not on this site (paper reading required)

• CSA2 entirely
• static mode assignment
• cross-layer reuse semantics (DeepSeek §2.2–2.3)

Mastery Check

• DERIVE: all three modes cold, tensor-by-tensor
• IMPLEMENT: 6-layer mixed-mode stack with per-layer accounting
• BENCHMARK: quality vs cache-usage Pareto
• DEBUG: a Reuse layer reading stale indices
• EXPLAIN: why reuse ≠ sharing routing everywhere

Part of CS/AI 684 — Efficient Long-Context Agent Systems.