Week 10 — Distributed Multimodal Training Infrastructure

Distributed · Pipeline Parallel · Systems

Advanced

III — Training the model

Overview

From model math to production training systems: comm/compute overlap, contrastive vision/text all-gather, disaggregated vision-encoder execution, long-sequence image sharding, context parallelism, shared state across pipeline stages, shadow indexers, microbatch lifetime, and sharded Engram storage. Cross-layer attention reuse is explicitly a distributed problem.

What You Will Learn

• Design a pipeline schedule for a CSA2 model split across devices.
• Compute bubble fraction and shared-state movement cost.
• Explain why an optimization is meaningless if its state cannot move.
• Reason about context parallelism and sharded conditional memory.

Core Concepts

Shared state across stages

A Reuse layer must materialize KV/indices produced on another pipeline stage; the schedule must order production before consumption.

Comm/compute overlap

Hidden state is the point: overlap communication with computation or the schedule stalls.

Prerequisites

Work these pages on this site before the lecture.

• Mixture of Experts
• Computational graphs
• ETL dependency orchestration

Lecture notes

CSA2 Reuse makes training a distributed problem: the layer that consumes KV/indices may live on another pipeline stage than the layer that produces them. If the schedule does not order production before consumption, you have a deadlock or a stale read — the same bug as week 4, now across devices.

Also in scope: context parallelism (shard the sequence), FSDP/Megatron-style parameter sharding, overlapping all-gather with compute, disaggregated vision encode, sharded Engram tables.

Your artifact is a schedule simulator (bubble fraction + state-movement bytes), not a real 8-GPU run.

Required readings

• DeepSeek-V4.1-Flash report — infrastructure
• Megatron-LM
• DeepSpeed documentation

Related on this site

• Computational graphs
• MoE chapter

Lab / Implementation

Design a pipeline schedule for a CSA2 model where Full and Reuse layers are split across devices; simulate it and compute bubble fraction plus shared-state movement cost.

Not on this site (paper reading required)

• context parallelism
• PP/FSDP mechanics
• shadow indexers
• disaggregated vision execution
• sharded Engram

Mastery Check

• DERIVE: bubble fraction and state-movement cost
• IMPLEMENT: CSA2 pipeline schedule simulator
• BENCHMARK: bubble fraction vs schedule
• DEBUG: a Reuse layer scheduled before its source state exists
• EXPLAIN: why shared state turns architecture into a distributed-systems problem

Part of CS/AI 684 — Efficient Long-Context Agent Systems.