A3 — Optimizer Comparison on a Large Embedding
Optimization · Sinkhorn
Hard
Prerequisites (this site)
Readings
Problem
Compare Adam-style, row-normalized and row+column-normalized (Sinkhorn) updates on a large embedding matrix.
Input
Log update RMS statistics per scheme and the downstream metric.
Output
A table of RMS + metric per scheme and a short note on when Sinkhorn wins.
Requirements
- Reproducible from a clean checkout.
- Reports a baseline and a metric with a confidence interval where applicable.
- Documents ≥2 expected failure modes and reproduces one.
Constraints
- No reference-implementation copying for the mechanism under test.
- Report where your method loses, not only where it wins.
Hints
- Fix all variables except the one you are sweeping.
- Instrument cost (FLOPs/bytes/time) before tuning quality.
- Write the honest limitation before writing the conclusion.
Assignment for CS/AI 684.