Skip to content
My Primers

Training: Overview — From FLOPs to a Finished Run

Training: From FLOPs to a Finished Run

Section titled “Training: From FLOPs to a Finished Run”

Every article in this folder is a chapter of one story: you are given a model target, a token budget, and a fleet — and you must turn chip-seconds into a trained model without wasting them. The chapters, in the order the reasoning actually happens:

  1. Sizing arithmetic — FLOPs ≈ 6·P·T, chips × MFU × time, 16 bytes/param. Currently lives inside the pod-training worked answer §2 and the mental-math drills; extraction into a standalone chapter is planned.
  2. Critical batch size — the optimizer ceiling: gradient noise scale, why B_crit grows during training, and how ceiling ÷ hardware floor caps useful fleet size.
  3. Memory — activation checkpointing (the compute↔memory dial) and ZeRO/FSDP (sharding model state). Gap: a mixed-precision-training chapter (bf16/fp8, master weights, loss scaling) is planned.
  4. Parallelism — the full stack (data/tensor/pipeline/sequence/context/expert) as the spine, with deeper cuts in context parallelism and 4D parallelism.
  5. Mixture of Experts — the parallelism specialization that dominates frontier training: routing, load balance, expert parallelism, and the failure modes.
  6. Data pipelines — a data platform for text-to-video training as the worked design. Gap: a general pretraining-data chapter (offline tokenization, packing, determinism, mixtures) is planned; the material exists in pod-training §4.
  7. Infrastructure and reliability — distributed training infrastructure (simplified version), plus rapid-fire Q&A. Gap: the goodput/checkpointing/SDC deep material lives in pod-training §5 and deserves its own chapter.

Capstone: the pod-scale MoE training worked answer applies every chapter above to one design question, and stays in the Google-interview track because its framing is interview-specific. The hardware-side counterpart of this whole folder is the JAX scaling book and the TPU/XLA article.

The through-line to keep while reading: arithmetic sets the shape (chapters 1–2), memory and communication set the layout (3–5), and data + reliability decide whether the layout’s throughput actually accumulates into a model (6–7).