A1 — Attention Resource Profile
Optimization · Attention
Medium
Prerequisites (this site)
Readings
Problem
Produce a compute/memory profile explaining when dense attention, sliding-window attention and sparse attention become dominated by different resources.
Input
For N in {512 … 32K}, report FLOPs, KV bytes, peak memory and wall-clock for each mode.
Output
A labeled plot with the crossover point per resource, plus a one-paragraph explanation.
Requirements
- Reproducible from a clean checkout.
- Reports a baseline and a metric with a confidence interval where applicable.
- Documents ≥2 expected failure modes and reproduces one.
Constraints
- No reference-implementation copying for the mechanism under test.
- Report where your method loses, not only where it wins.
Hints
- Fix all variables except the one you are sweeping.
- Instrument cost (FLOPs/bytes/time) before tuning quality.
- Write the honest limitation before writing the conclusion.
Assignment for CS/AI 684.