Week 1 — Why Agent Workloads Change Model Architecture

Long Context · Systems · Inference

Advanced

I — Long-context architecture

Overview

Start with the workload, not the Transformer. Long-horizon agents make repeated tool calls over input-heavy contexts; that shifts the bottleneck from arithmetic to storage and data movement. After sparse attention reduces compute, cache design becomes an architectural concern rather than an inference-only optimization.

What You Will Learn

• Separate prefill from decode cost and say which dominates each workload.
• Place KV state on the memory hierarchy (HBM → DRAM → SSD) and name the bandwidth limit.
• Explain why local attention alone cannot carry global agent state.
• Distinguish relational/path retrieval from semantic similarity.

Core Concepts

Prefill vs decode

Prefill is compute-bound and parallel over the prompt; decode is memory-bandwidth-bound and serial. Agent sessions are input-heavy, so prefill and prefix reuse dominate the bill.

The memory hierarchy is the real constraint

Once sparse attention cuts FLOPs, the limits become cache size, HBM bandwidth, host memory and SSD movement. Optimizations that ignore where state lives are meaningless.

Graph retrieval vs vector retrieval

Vector similarity returns semantically related chunks but does not represent causal or relational paths. Facts as subject–relation–object triples let you retrieve connected subgraphs instead.

Prerequisites

Work these pages on this site before the lecture.

• KV cache
• BM25
• Streaming min-max
• Matrix multiplication

Lecture notes

An agent session is not a chatbot turn. The prompt is large (tools, traces, retrieved docs), decode is a long tail of tool-call tokens, and the same prefix is reused across steps. That is why prefill (parallel, compute-bound) and decode (serial, bandwidth-bound) have to be costed separately.

KV bytes for a transformer-style cache:

\mathrm{KV\ bytes}\approx 2\cdot B\cdot L\cdot H\cdot D\cdot S\cdot b

where the leading 2 is keys and values, B batch, L cached layers, H KV heads, D head dim, S sequence, b bytes per element. Missing the \times 2 or the batch factor is the usual “my VRAM estimate is half of reality” bug.

After sparse attention cuts FLOPs, the remaining limits are where the state lives: HBM (fast, small), host DRAM, SSD. PagedAttention (vLLM) treats the cache like virtual memory so fragments can be shared across requests. Your week-1 artifact is a cost model, not a kernel.

Graph retrieval is in the course from day one because vector similarity returns nearby text, not a path of facts. You will build that system in weeks 13–14; this week you only need the distinction.

Required readings

• DeepSeek-V4.1-Flash technical report (PDF) — primary text for the whole course
• DeepSeek-V4.1-Flash model card
• vLLM documentation and PagedAttention (arXiv:2309.06180)
• Roofline model (Williams et al., 2008)

Related on this site

• KV cache
• BM25
• Week 2 — attention economics

Lab / Implementation

Build a KV-cache cost model sweeping context 4K→1M; classify which resource binds for four workloads (short chat, huge prompt, long agent session, repeated prefix reuse).

Not on this site (paper reading required)

• HBM/DRAM/SSD bandwidth figures
• arithmetic intensity / roofline
• PagedAttention, vLLM

Mastery Check

• DERIVE: the KV bytes formula and the prefill/decode split
• IMPLEMENT: the cost model + binding-resource classifier
• BENCHMARK: predicted vs measured peak VRAM
• DEBUG: a mismatch caused by the missing ×2 or batch factor
• EXPLAIN: where memory lives in a 1M-token agent session

Part of CS/AI 684 — Efficient Long-Context Agent Systems.