Week 12 — Asynchronous RL, Reasoning Effort and Agent Infrastructure
RL · Async · Systems
Advanced
IV — Agent post-training
Overview
Long-tail rollout latency, asynchronous generation, sample-level dispatch, off-policy data, length bias, staleness masking, checkpoint preemption, token-level interruption, persisted KV/routing state. RL is conditioned on a scalar reasoning-effort variable b with effort-dependent length penalties, letting one checkpoint occupy a cost/quality frontier.
What You Will Learn
- Derive the length bias from first-completed-first-trained dispatch.
- Explain staleness masking and what it trades away.
- Implement effort-conditioned length penalties.
- Reason about DSec-style sandboxed execution.
Core Concepts
Effort-conditioned penalty
k(b)=k_0\exp\!\left(-\frac{b-b_{\min}}{\tau}\right)
Length bias
If the first completed samples enter training first, short rollouts are over-represented; staleness masking corrects the resulting distribution shift.
DSec
Sharded infrastructure with relaxed global consistency, local admission constraints, high-density container execution and sandbox protections.
Prerequisites
Work these pages on this site before the lecture.
Lecture notes
Long-tail rollouts make synchronous RL wasteful. Async dispatch trains the first-finished samples first, which over-represents short traces (length bias). Staleness masking throws away or down-weights off-policy leftovers; over-masking deletes the on-policy signal.
Effort b is a conditioned scalar: one checkpoint, many cost/quality points via
k(b)=k_0\exp\!\left(-\frac{b-b_{\min}}{\tau}\right)
DSec is the sandboxed execution fabric (containers, local admission, relaxed global consistency). You simulate the queue; you do not build a cluster.
Required readings
- DeepSeek-V4.1-Flash report § async RL / effort
- PPO
- vLLM serving docs (prefix cache / continuous batching as the inference half)
Related on this site
Lab / Implementation
Simulate an async rollout queue and quantify the length bias; add staleness masking and measure the correction; implement a k(b) effort penalty and plot the cost/quality frontier.
Not on this site (paper reading required)
- async RL mechanics
- staleness masking
- checkpoint preemption
- DSec (DeepSeek §5)
Mastery Check
- DERIVE: length bias and the k(b) penalty
- IMPLEMENT: async queue + masking + effort conditioning
- BENCHMARK: bias magnitude; cost/quality frontier
- DEBUG: over-aggressive masking dropping on-policy samples
- EXPLAIN: when async is not worth it