Week 8 — MoE and Multimodal Architecture

MoE · Multimodal · Routing

Advanced

III — Training the model

Overview

Routed vs shared experts, modality-specific expert-balancing biases, multimodal token distributions, a 32-layer vision encoder, 2D-RoPE and 3×3 pixel unshuffle (a 9× token reduction). Ordinary global balancing can hide modality imbalance.

What You Will Learn

• Implement top-k MoE routing and the auxiliary load-balancing loss.
• Explain capacity, overflow and expert collapse.
• Implement pixel unshuffle and 2D positional handling.
• Show how global balancing hides modality starvation.

Core Concepts

Top-k routing

y=\sum_{i\in \text{Top-}k} g_i(x)\,E_i(x),\qquad g=\operatorname{softmax}(\text{gate}(x))

Pixel unshuffle

3\times 3\ \text{unshuffle}\Rightarrow \text{visual tokens}\div 9

Modality imbalance

Image and text tokens have different routing distributions; a single global balance target can look healthy while one modality starves.

Prerequisites

Work these pages on this site before the lecture.

• Mixture of Experts
• Softmax
• 2D convolution
• Bilinear interpolation

Lecture notes

Routed experts + a shared expert is the DeepSeek/GLM pattern you already met on the research track. Multimodal adds a second token population: vision tokens after a 3×3 pixel unshuffle (9× fewer tokens) and 2D-RoPE.

A single global load-balancing loss can look healthy while one modality starves. Your lab must construct that case (two modalities, one unused expert cluster) and then add per-modality bias/correction.

Required readings

• DeepSeek-V4.1-Flash report — MoE + vision
• Switch Transformers
• ViT
• Mixture of Experts chapter

Related on this site

• DeepSeek MoE router
• Gemma 3 local vs global routing

Lab / Implementation

Build a top-k MoE router with separate load-balancing corrections for two modalities; construct a case where global balancing masks imbalance, then fix it and measure utilization entropy.

Not on this site (paper reading required)

• vision encoder
• 2D-RoPE
• pixel unshuffle
• multimodal token mixture

Mastery Check

• DERIVE: routing math, aux loss, capacity/overflow
• IMPLEMENT: two-modality MoE + pixel unshuffle
• BENCHMARK: utilization entropy, tokens/quality
• DEBUG: expert collapse / overflow
• EXPLAIN: why global balancing hides modality starvation

Part of CS/AI 684 — Efficient Long-Context Agent Systems.