GPU trace-reading drills

A synthetic Nsight-style timeline: eleven scenario types, each with several framings — including lookalike pairs and traces with nothing wrong at all. Read it out loud in the fixed order, commit to a diagnosis, then reveal.

The reading order — always the same

  1. Find the step boundary. What's the step time, and what should it be?
  2. Gap analysis: step time − sum of GPU busy time. Where do the gaps line up?
  3. Overlap: is communication hidden behind compute, or serialized after it?
  4. Longest kernels: name the top one; is its duration justified by its math?
  5. Only then zoom into a single kernel (that's Nsight Compute's job, not the timeline's).

Trace

2:00

Press New trace to start.

GPU compute NCCL / comm CPU / CUDA API memcpy / alloc

Say: what's the step time, where are the gaps, what's your diagnosis, what one measurement confirms it, what's the fix and the expected win.