Week 7 — mHC, Engram and DSpark
Conditional Memory · Speculative Decoding
Advanced
II — Memory-efficient inference
Overview
Three separate directions. Single-Pass mHC reorganizes residual-stream mixing to hit the ideal activation-traffic bound. Engram is conditional memory: 196B parameters of sparsely accessed hashed N-gram embeddings, decoupled from computation. DSpark is speculative decoding with a Markov head and confidence-scheduled verification.
What You Will Learn
- Separate parameters used for computation from parameters used as addressable memory.
- Model speculative throughput as a function of acceptance probability and draft length.
- Build an Engram-style hashed conditional-memory layer.
- State the exact speculative-decoding acceptance rule.
Core Concepts
Conditional memory
Engram replaces some computation with addressable memory: hash N-grams → sparse embedding lookup → gate.
Speculative throughput
\mathbb E[\text{tokens/round}]=f(\text{acceptance},\ \text{draft length})
DSpark
Semi-autoregressive drafting + a Markov head + confidence-scheduled verification; trained after backbone pretraining and kept aligned during post-training.
Prerequisites
Work these pages on this site before the lecture.
Lecture notes
Three mechanisms, three jobs:
- mHC — residual-stream mixing that hits an activation-traffic bound (data movement, not a new attention op).
- Engram — hashed N-gram embeddings used as addressable memory, not as another dense FFN. Parameters that are rarely touched should not sit in the matmul path.
- DSpark — speculative decoding with a Markov draft head and confidence-scheduled verification. Throughput is f(\text{acceptance},\ \text{draft length}). If the verifier changes the distribution, you are no longer doing speculative decoding.
Required readings
- DeepSeek-V4.1-Flash report — mHC / Engram / DSpark
- Fast inference from transformers via speculative decoding
- Speculative decoding chapter
Related on this site
- LoRA (low-rank / extra-parameter intuition)
Lab / Implementation
Model speculative throughput vs acceptance × draft length and validate it; build a tiny Engram layer and compare against an equal-parameter dense FFN.
Not on this site (paper reading required)
- mHC
- Engram
- DSpark (Markov head, confidence-scheduled verification)
Mastery Check
- DERIVE: the exact acceptance rule and expected-tokens model
- IMPLEMENT: speculative sampling + Engram layer
- BENCHMARK: throughput vs acceptance; quality/param vs FFN
- DEBUG: a verifier that changes the distribution
- EXPLAIN: when to spend parameters on memory vs computation