Fleet-Wide ML Performance Regression Detection

A staff-level model answer, written out in full. Primary fleet: TPU (v5e / Trillium / Ironwood, XLA weekly releases). Secondary: GPU (H100 / B200, CUDA/driver/TRT-LLM matrix). All arithmetic uses the worked-example config defined in §7 and is consistent across sections.

1 · Clarifying Questions, Requirements, Capacity Math

The questions I ask before designing anything, each with the assumption I proceed on:

Question	Why it changes the design	Assumed answer
Are we gating releases (compiler) or models (retrains)?	Gating axis determines what a "series" is and who gets paged	Both, but compiler releases are the primary gate; model retrains reuse the same rails
What rolls out weekly?	Sets detection-latency budget	XLA release cut Monday, prod rollout Thursday → 72 h budget end-to-end; nightlies get a smoke signal in < 4 h
Fleet composition?	Variance model and version matrix differ radically	~85% TPU (v5e legacy, Trillium bulk, Ironwood ramping), ~15% GPU (H100 → B200). GPU adds a 3-axis version matrix (driver × CUDA × serving stack) that TPU doesn't have
How many models matter?	Coverage strategy	~300 production models; top 20 ≈ 80% of fleet spend (spend-weighted coverage, not model-count coverage)
What is a regression in dollars?	Sets the minimum detectable effect worth building for	1% fleet-wide cost/token ≈ order $10M/yr — 1% is the target MDE on the top tier, not 5%
Who consumes verdicts?	Attribution is a product with two customers	Compiler team (needs culprit CL + HLO diff) and model teams (needs "your model, this config, this much, here's the trace")

Functional requirements: detect regressions in TTFT, TPOT, goodput, MFU, HBM-BW utilization, cost/token across model × platform × config series; attribute to culprit commit (compiler axis) or config delta (hardware axis); gate releases; track improvements (§9) as first-class.

SLOs for the detection system itself — a detector without its own SLOs becomes an unowned dashboard:

SLO	Target	Rationale
Detection latency (release-blocking tier)	< 24 h from RC cut	Leaves 48 h of the 72 h window for triage + fix/rollback decision
Detection latency (smoke tier)	< 4 h from nightly	Narrows any later bisection to ~1 day of commits
False-page budget	≤ 5 pages/week fleet-wide	One triage rotation absorbs this; beyond it, gates get ignored (§8)
Coverage	≥ 95% of fleet spend in T1; 100% of models monthly	Spend-weighted; the long tail rotates (§5)
MDE @ power 0.9	1% (top-20), 3% (tail)	Derived from the dollar value of 1% and the repeat budget (§4)

The capacity math that forces the design

Naive full matrix, per sweep:
  300 models × 4 platforms (v5e, v6e, Ironwood, B200) × 6 configs
  (2 batch regimes × {bf16,int8/fp8} × {short,long ctx})     = 7,200 points
  × 5 repeats × 15 min                                        = 9,000 chip-hours/sweep
Release cadence: 7 nightlies + 1 RC = 8 sweeps/week           = 72,000 chip-hours/week
                                                              ≈ 429 chips running 24/7

That is a dedicated mid-size pod doing nothing but benchmarking — order $15–25M/yr of hardware — and it still doesn't cover new-silicon qualification. Nobody funds that, and nobody should: the matrix is massively redundant (most models are HLO-similar to each other, most nightlies change nothing perf-relevant). The design problem is therefore coverage per chip-hour, which is what the tiering in §5 optimizes. Target budget: ~4,500 chip-hours/week, ~6% of naive, with ≥95% spend-weighted coverage.

Staff signal: leading with the cost/coverage capacity math — and with SLOs for the detector itself including a false-page budget — frames the whole problem as an engineering-economics design rather than "run benchmarks, alert on diffs." L5 answers start drawing the pipeline; L6 answers first prove the naive pipeline is unaffordable and derive what any affordable one must do.

2 · High-Level Architecture

Key placements, and why: pre-submit gets no chips — per-commit hardware benchmarking of a compiler is unaffordable and unnecessary; pre-submit runs static checks only (HLO cost-model deltas, fusion-count diffs, roofline estimates on canonical HLO — flags "this CL changes generated code for N benchmarks" so the nightly knows where to look). Nightly T0 exists to keep bisection ranges small (~1 day ≈ 30–60 CLs), not to gate. T1 is the release gate and owns the 24 h SLO. Prod canary (§8) is the backstop for what benchmarks structurally can't see: real traffic mixes, real cache hit rates, real interference. Every result row is keyed by (model-rev, HLO fingerprint, compiler CL, platform, config, firmware) — the fingerprint is what makes attribution and dedup possible later.

Staff signal: refusing to put chips in pre-submit — and saying why in cost terms — plus using the nightly tier explicitly as a bisection-range compressor rather than a gate, shows the tiers were derived from the capacity math and the triage workflow, not copied from a CI diagram.

3 · Benchmark Suite Design

3.1 Workload selection and stratification

The suite is stratified so that every axis that changes the bottleneck is represented, because a regression only reproduces in the regime where its resource is binding:

Stratum	Axis values	Why it's a stratum
Phase regime	prefill-heavy (8K/32K prompt) vs decode-heavy (B∈{8,64}, long-lived)	Compute-bound vs memory-bound: the same code change can be invisible in one and 8% in the other (§6's example)
Architecture	dense 8B/70B-class · MoE (fine-grained, top-8) · embedding-heavy ranking (SparseCore path)	Different op mixes exercise different compiler passes (fusion, all-to-all layout, embedding lowering)
Numerics	bf16 · int8 (AQT) · fp8 (Ironwood/B200 only)	Quantized paths have their own kernels and their own regressions
Geometry	MXU-aligned shapes AND deliberately misaligned ones (head-dim 64, odd vocab)	See below — measuring the hardware vs measuring a mismatch
Scale	single-chip · slice (ICI collectives in path) · multislice (DCN)	Collective regressions never reproduce on one chip

The geometry-honesty point. A head-dim-64 model on Trillium's 256×256 MXU pads every attention GEMM tile 4× — it runs at ≤25% of the MXU by construction. If that model is in the suite as a hardware benchmark, it reports "Trillium is slow," which is false; the truth is "this shape wastes Trillium." The suite must carry both, labeled: geometry-matched variants measure hardware capability (regressions here are compiler/hardware bugs); production-shaped variants measure delivered performance (regressions here may be padding/layout decisions — still real dollars, different owner). Conflating the two poisons cross-hardware claims (§9) and misroutes pages.

3.2 Metrics — precise definitions, and how they lie

Metric	Definition (exact)	How it silently lies
TTFT	request admitted → first output token committed; report p50/p99 per config	Queue-wait exclusion hides scheduler regressions; include it, and also report compute-only prefill time separately
TPOT	steady-state decode: (t_last − t_first)/(n−1) at fixed resident batch	Measured at unspecified batch, it's meaningless — batch is half the roofline. Every TPOT is (TPOT \| B, s̄)
Goodput	tokens/s/chip while p99 TPOT ≤ SLO and p99 TTFT ≤ SLO, at the arrival rate that maximizes it	Throughput without the SLO clause rewards batch inflation that ships unusable latency
MFU	model FLOPs (analytic, fwd 2·P_active/token + exact attention term) ÷ (peak dense FLOPS of the claimed dtype × chips × wall time)	Three classic lies: counting rematerialization FLOPs in the numerator (inflates training MFU 20–30%); forgetting the causal ½ on attention (inflates the attention term 2×); MoE accounted at total instead of activated params (inflates 10×). Fix: numerator is useful algorithmic work only, defined once in code, shared by every consumer
HBM BW util	counter-measured bytes/s ÷ platform peak (per chip, averaged over the measurement window)	DMA prefetch double-counting; window boundaries clipping bursts. Use the hardware counters, not analytic bytes
Cost/token	(chips × internal $/chip-hr) ÷ goodput — at the goodput operating point	Priced at max throughput instead of the SLO point; and internal vs external $/chip-hr changes every cross-platform ranking — state the basis

3.3 Load generation

Open-loop (Poisson arrivals at fixed rate λ, timestamps independent of completions) for anything with an SLO in it: closed-loop load generators wait for a response before sending the next request, so when the system slows, offered load drops — the generator colludes with the system under test to hide tail latency (coordinated omission). Closed-loop is kept only for max-throughput ceiling measurement, labeled as such. Arrival mix: trace-derived length distributions per model (p50/p95/p99 input and output lengths sampled from production logs, refreshed monthly) — synthetic uniform lengths miss both the prefix-cache and the long-tail-KV behavior that decide real goodput. Warm-up is discarded (first 2 min: compile cache fill, allocator steady-state, cache warm), and each measurement window is long enough for ≥ 10× the longest request.

Staff signal: the MFU-accounting-lies table and the geometry-matched-vs-production-shaped split are benchmark honesty engineering — the difference between a suite that measures the fleet and one that generates plausible numbers. L5 lists metrics; L6 defines each one's denominator and names the three ways it gets gamed, because the detector's verdicts are only as trustworthy as the metric definitions.

Interviewer follow-ups (§3)
Q: Your benchmark models drift from production models — retrains change shapes quarterly. How does the suite stay representative?
A: The suite is keyed by HLO fingerprint, not model name. A weekly job fingerprints every production model's serving HLO, clusters by op/fusion signature, and diffs cluster membership against the suite; when a cluster's representative drifts past a similarity threshold, the suite auto-adopts the new HLO (with its production trace-derived load mix) and re-baselines that series. Baselines are never carried across fingerprint changes — that's how you avoid "regression" alerts that are actually model changes.
Q: Why not just benchmark on production replicas with mirrored traffic and skip the synthetic suite?
A: Mirrored traffic is the canary tier (§8), and it's essential — but it can't gate releases: you can't run 8 compiler versions against production simultaneously, you can't hold traffic constant across a week, and you can't reproduce a measurement for bisection. The synthetic suite exists to make the input a controlled variable; the canary exists to catch what control misses. You need both, and each one's weakness is the other's job.
Q: How do you benchmark the prefix-cache-dependent goodput honestly?
A: Replay length-and-prefix-structured traces (session trees sampled from production, anonymized), report goodput at the production-observed hit rate and at hit=0 as a bracket. A compiler change can't move the hit rate, so for release gating the hit=0 series is the sensitive one; the bracketed pair exists so capacity planning doesn't misread the gate numbers.

4 · The Statistics of Regression Detection

4.1 Variance sources and realistic magnitudes

Source	TPU magnitude	GPU magnitude	Control
Run-to-run, same chip, pinned everything	0.2–0.6% (deterministic execution, AOT compile)	1.5–4% (DVFS, clock/thermal state)	GPU: lock clocks (`nvidia-smi -lgc`), fixed power limit, thermal soak before measuring
Host jitter (input feed, RPC)	0.3–1%	0.5–2%	Dedicated hosts, pinned CPUs, warm-up discard
Cell/chip heterogeneity (binning, HBM vendor)	0.5–1.5% across chips	1–3%	Benchmark on a reserved, fixed set of chips; treat chip-set change as a baseline reset
ICI/NVLink congestion from neighbors	0–2% if shared	0–3%	Reserved sub-pods; never benchmark on scavenger capacity

Net, with controls: σ ≈ 0.4% on TPU, σ ≈ 2% on GPU for latency-class metrics. This single asymmetry drives different repeat counts, different MDEs, and is an honest reason TPU fleets are easier to regression-test — determinism is an operability feature.

4.2 Noise floors, empirically

Per (model, platform, config, metric) series: 30 runs across ≥ 5 days and ≥ 3 chip-sets with no software changes establish the null distribution. Store median and MAD (robust to the occasional infra hiccup); the noise band is ±3·MAD_normalized. Re-derive quarterly and on any executor-fleet change. Series whose empirical σ exceeds 2× the platform norm get quarantined and investigated as infrastructure bugs — a noisy series is a defect of the harness, not a fact of nature to alert around.

4.3 Detection method — chosen, with reasons

Two consumers, two methods:

(a) Release gate (T1): two-sample comparison of RC vs current-prod release, n repeats each. Welch/Mann-Whitney p-value plus a minimum-effect filter: fire only if p < α_adj and observed Δ > max(MDE_tier, 2σ). The effect-size filter is what keeps statistically-significant-but-economically-irrelevant 0.3% wiggles out of the gate.

(b) Trend monitoring (nightly series): changepoint detection — E-divisive / PELT-class — on each series. Why changepoint over fixed thresholds: it localizes the day the level shifted (directly seeding bisection), composes evidence across noisy points instead of judging each nightly alone, and survives slow drift + step mixtures. Why not pure sequential tests (CUSUM per point): they're optimal for a known shift size, but we care about unknown sizes and want the localization for free.

4.4 Multiple comparisons → false-page budget → on-call load

Series under gate: 60 models × 2 platforms × 3 configs × 6 metrics ≈ 2,160 tests per RC
Naive α = 0.05                    → ~108 expected false positives per release. Untriageable.
Benjamini–Hochberg FDR at 5% on the gated family, plus the 2σ effect filter:
  expected false pages ≈ 0.05 × (true alerts, typically 5–15) ≈ < 1/RC
Nightly trend family (~4,000 series): changepoint alerts routed to a queue, not a pager;
only spend-weighted top-20 series page directly.
Result: ≈ 3–5 pages/week — matches the §1 budget and one triage rotation.

4.5 Statistical power — choosing n per tier

n per arm to detect Δ at significance α=0.05, power 0.9 (z_α/2=1.96, z_β=1.28):
  n ≈ 2 · (1.96+1.28)² · (σ/Δ)²  =  21 · (σ/Δ)²

TPU, σ=0.4%:  Δ=1% → n ≈ 3.4 → 5 repeats (margin for discard)   [T1 top-20]
              Δ=3% → n ≈ 1   → 3 repeats                         [T1 tail / T0]
GPU, σ=2%:    Δ=1% → n ≈ 84  → unaffordable → either pin clocks harder,
              or accept MDE=4% at n=6 and let the prod canary own the 1–4% band.

That last line is a real design decision stated honestly: on GPU, small regressions are caught in canary (big N from live traffic), not in benchmarks — and the alert docs say so, so nobody believes GPU benchmarks are certifying 1%.

Staff signal: mapping α → expected pages → one on-call rotation makes the false-positive budget an organizational design constraint, and the power calculation makes n a derived quantity instead of folklore ("we run 5 times"). The honest GPU concession — benchmarks certify 4%, canary owns 1–4% — is the kind of scoping an L6 states unprompted.

Interviewer follow-ups (§4)
Q: Your executor fleet gets recabled and every baseline shifts 0.7%. What happens Monday?
A: Fleet-change events are first-class inputs: any executor-fleet mutation emits a baseline-invalidation event that freezes gating on affected series, triggers a 30-run re-baseline, and annotates dashboards so the step isn't read as a compiler event. The detector must subscribe to its own infrastructure's change feed — otherwise its first serious incident is self-inflicted.
Q: A regression lands in the same week as an intentional improvement of similar size in the same series. Does changepoint detection see it?
A: A single series can net them out — this is why the gate compares against expected deltas: compiler teams register intended wins with the RC (per-benchmark expected Δ from their own runs); the gate tests observed − expected against zero. Unregistered movement in either direction fires. Improvement tracking isn't a courtesy; it's what keeps the null honest (§9).
Q: Why MAD, not standard deviation?
A: Benchmark null distributions are contaminated: rare infra hiccups (a slow host, a retried RPC) put 5–10σ outliers in the tail at ~1% rates. Standard deviation lets a handful of those inflate the band and blind the detector; MAD ignores them, and the outliers get their own infra-health alert instead of widening every noise band.

5 · Tiered Execution Strategy

Weekly budget:  T0: 7 × 40 h              =   280 chip-h
                T1: 1 RC × 1,350 h        = 1,350
                T2 amortized (2,300/4)    ≈   575
                tail rotation (75 models × 1 platform × 2 cfg × 3 reps × 15 m) ≈ 1,150
                re-baselines + bisection reserve                              ≈ 1,100
                total ≈ 4,455 chip-h/week ≈ 6% of the naive 72,000            ✔ §1 target

Long-tail sampling. The 240 non-T1 models rotate through weekly quarters (every model measured monthly), plus HLO-fingerprint clustering: models whose serving HLO is ≥95% signature-similar to a T1 representative inherit its verdict and skip redundant chip time. Coverage is spend-weighted; the report says "95% of fleet dollars gated, 100% of models within 28 days," never "20% of models tested."

Pruning with cost models. Pre-submit static analysis (per-op roofline over the analytic bytes/FLOPs of each benchmark's HLO, per platform) marks each CL as perf-relevant or not per benchmark; nightlies skip benchmarks no CL since the last run could have touched (typical nightly: 60–80% skipped, reclaimed as bisection reserve). The same cost model provides pre-silicon estimates for §9's hardware-qualification and sanity-bounds every measured result (measured ≪ roofline prediction with no HLO change = suspect the harness first).

Staff signal: the pyramid's budget line reconciling to the §1 target (4,455 ≈ 6% of naive) closes the loop the capacity math opened — and HLO-fingerprint inheritance is the insight that coverage is a property of programs, not model names: benchmarking two models that compile to the same kernels is paying twice for one answer.

6 · Root-Cause Attribution

Attribution is a product with two customers: the compiler team needs which CL and which pass; the model team needs which op, how much, and what to do meanwhile. The system produces both from the same artifacts.

6.1 Automated bisection — compiler axis

6.2 Drill-down artifacts

Every T1/T0 run persists: per-op timing from the profiler (op name → µs, keyed to HLO), the serialized post-optimization HLO (so two compiler versions' decisions can be diffed, not just their outputs), kernel-level traces for the top-N ops, and the collective timeline (per-collective start/duration/bytes). The attribution job's first move on any alert is a per-op diff: join the before/after op-timing tables on HLO fingerprint, sort by Δµs — in practice this localizes 80% of regressions to a named op in minutes, before any human looks (Artifact 6 in §7 is exactly this table for the worked example).

6.3 Worked example, end to end

Setup (the numbers every artifact below uses): model chat-8b: 8.0B params, int8 weights (8 GB), 32 layers, d_model 4096, 32 Q heads / 8 KV heads × head-dim 128, int8 KV. Served on a v6e-4 (Trillium) slice: 4 × 1.64 TB/s = 6.56 TB/s aggregate HBM BW; measured achieved fraction 0.65 → 4.26 TB/s. Decode config: B=64, mean context s̄=2048.

Baseline decode step (per token, whole model):
  weights streamed:  8 GB
  KV read: 2·L·H_kv·d·s̄·B·1B = 2·32·8·128·2048·64 ≈ 8.6 GB
  total ≈ 16.6 GB  ÷ 4.26 TB/s = 3.90 ms TPOT          → per layer ≈ 122 µs
  per layer: GEMM block (250 MB weights → 59 µs) + fused attention (269 MB KV → 63 µs)

The event. XLA RC 2026-06-17 raises a T1 alert: chat-8b · v6e-4 · decode B=64 · TPOT p50: 3.90 → 4.22 ms (+8.2%). Same model on v5e-8: no movement. Prefill TTFT: no movement. The nightly changepoint (§4 plot) already pinned the day; the cost-model filter cut 38 CLs to 9; bisection (4 builds) lands on cl/7841917 — "fusion: cap producer-consumer fusion VMEM footprint at 1.5 MB to prevent double-buffering spills."

The trace comparison (§7 renders both): before — one kernel fusion.1423 (attention_softmax_fused), 63 µs/layer, scores never leave VMEM. After — fusion.1423a (qk_scores) 34 µs + softmax.88 on the VPU 8 µs + fusion.1423b (attn_v) 37 µs, and a new HBM round trip: the fp32 score matrix S (B·H·s̄·4 B = 64·32·2048·4 ≈ 16.8 MB/layer) written then re-read: 33.6 MB/layer of new traffic.

Roofline check (why decode regresses):
  new traffic: 33.6 MB × 32 layers = 1.07 GB/step ÷ 4.26 TB/s = +252 µs  → +6.5% predicted
  measured +8.2% (+320 µs): remainder ≈ exposed VPU softmax (8 µs × 32 = 256 µs *partially*
  overlapped → ~60 µs exposed) + lost intra-kernel pipelining. Predicted-vs-measured gap
  reported honestly; the roofline term is the dominant, actionable component.
Why v5e is unaffected:  per-core score tile = lanes × s̄ × 4B → v6e (256 lanes): 2.0 MB > 1.5 MB cap → split
                                                v5e (128 lanes): 1.0 MB < cap → stays fused
Why prefill is unaffected: prefill attention is compute-bound (AI ≫ ridge); the added bytes
  hide under MXU time — and flash-style prefill never materialized S to HBM anyway.

Remediation paths, with tradeoffs: (1) Compiler flag now: allowlist --xla_fusion_vmem_cap_exempt=attention_softmax on affected models — instant, reversible, adds flag-debt that must be tracked to removal. (2) Heuristic fix: make the cap tile-geometry-aware (footprint ÷ lanes, not absolute bytes) — correct fix, lands next release, needs the compiler team. (3) Model-side: none reasonable here; reshaping heads to duck a compiler cap is backwards. Gate verdict: block RC for the affected family, ship with (1) as the documented exception, track (2) to closure.

Staff signal: the worked example runs the full loop — alert → per-op diff → HLO diff → roofline arithmetic that quantitatively explains platform- and phase-selectivity (+6.5% predicted of +8.2% measured, v5e exempt by tile math) → three remediations with owners and debts. Attribution designed as a two-customer product, and the predicted-vs-measured gap reported instead of massaged, are the L6 tells.

Interviewer follow-ups (§6)
Q: The culprit CL is a giant LLVM roll (10,000 upstream commits). Bisection says "the roll." Now what?
A: Rolls get their own inner bisection lane: we maintain the ability to build XLA at synthetic points inside a roll (upstream repo bisect driven by the same harness), budgeted separately because compile time dominates. In parallel, the per-op diff usually names the op and pass; that plus the roll's changelog often localizes without full inner bisection. If neither converges within the SLO, the gate doesn't wait: block-or-flag decides on the measured effect, and root-cause continues out of band.
Q: Two CLs interact — each is neutral alone, together they regress. Bisection returns an innocent CL. How do you catch this?
A: Bisection verifies its verdict: after landing on CL k, it re-tests k−1 (must be clean) and k alone cherry-picked onto the pre-range base (must reproduce). Interaction cases fail that second check, which auto-escalates to a bounded pairwise search over the cost-model-flagged subset — 9 flagged CLs is 36 pairs, still cheap at 3 reps. The verification step is what keeps trust in the bisector's verdicts.
Q: The regression only reproduces at B=64, and your bisection harness runs B=8 for speed. Comment.
A: Bisection must reproduce the alerting series' exact config — the alert payload carries the config hash precisely so the bisector can't "optimize" itself into a different regime. Roofline thinking says why: bottleneck-dependent regressions (this one: memory-bound only) vanish outside their regime. Any speedup in bisection comes from fewer repeats, never from changing the workload.

7 · Trace and Log Artifacts

All artifacts share the §6.3 config: chat-8b (8 GB int8, 32 layers, 32Q/8KV × 128, int8 KV) on v6e-4 @ 4.26 TB/s achieved; B=64, s̄=2048; one decode layer ≈ 122 µs before, ≈ 132 µs after. Timelines show one layer window.

Artifact 1 — Perfetto-style timeline, BEFORE (xla @ prod): one decode layer, 122 µs

Artifact 2 — Perfetto-style timeline, AFTER (xla @ RC-2026-06-17): one decode layer, 132 µs

Artifact 3 — alert payload (detector → triage queue)

{
  "alert_id": "perfreg-2026-06-18-00412",
  "series": {"model": "chat-8b", "hlo_fingerprint": "hlo:9f31c2…",
             "platform": "v6e-4", "config": "decode.B64.s2048.int8", "metric": "tpot_p50_ms"},
  "baseline": {"median": 3.90, "mad": 0.017, "n": 30, "window": "2026-05-12..06-16"},
  "observed": {"median": 4.22, "n": 5, "release": "xla-rc-2026-06-17"},
  "delta": {"abs_ms": 0.32, "pct": 8.2, "z_robust": 18.8, "fdr_q": 1.2e-9},
  "effect_gate": {"mde_pct": 1.0, "passed_min_effect": true},
  "scope": {"correlated_series": ["chat-8b/v6e-4/decode.B8: +7.4%",
            "rank-moe-110b/v6e-16/decode: +3.1%"],
            "clean_series": ["chat-8b/v5e-8/*", "chat-8b/*/prefill.*"]},
  "changepoint": {"nightly": "2026-06-17", "cl_range": ["cl/7841885", "cl/7841923"], "n_cls": 38},
  "bisection": {"status": "done", "culprit": "cl/7841917", "verified_adjacent_clean": true},
  "artifacts": {"op_diff": "…/opdiff/00412", "hlo_diff": "…/hlodiff/00412",
                "trace_before": "…/tr/prod/9f31c2", "trace_after": "…/tr/rc/9f31c2"},
  "verdict": "BLOCKING", "owner_hint": "xla-fusion", "page": "ml-perf-triage"
}

Artifact 4 — bisection log (excerpt)

[00:07] range cl/7841885..cl/7841923 (38 CLs); cost-model filter → 9 perf-relevant
[00:09] build xla @ cl/7841904 … TPOT 3.91 ±0.02  → CLEAN   range → (7841904, 7841923], 4 CLs
[00:31] build xla @ cl/7841917 … TPOT 4.21 ±0.03  → REGRESSED range → (7841904, 7841917], 2 CLs
[00:52] build xla @ cl/7841911 … TPOT 3.90 ±0.02  → CLEAN   range → (7841911, 7841917], 1 CL
[01:13] candidate cl/7841923? no — remaining: cl/7841917 "fusion: cap producer-consumer
        fusion VMEM footprint at 1.5MB" … wait: re-map — flagged CLs in range: {7841917}
[01:14] verify: cl/7841916 CLEAN ✔ ; cherry-pick 7841917 on base → 4.22 REPRODUCED ✔
[01:35] culprit = cl/7841917 (author: xla-fusion team) · confidence: verified-both-sides

Artifact 5 — HLO diff (post-optimization, abbreviated)

--- prod   hlo:9f31c2  layer.attn
-  %fusion.1423 = bf16[64,32,128]{...} fusion(%q, %k_cache, %v_cache),
-      kind=kCustom, calls=%attention_softmax_fused_computation
-      backend_config={"fusion_kind":"attention_softmax_fused","vmem_bytes":2097152}
+++ rc-2026-06-17
+  %fusion.1423a = f32[64,32,2048]{...} fusion(%q, %k_cache), kind=kLoop,
+      calls=%qk_scale_computation            // S materialized to HBM (16.8 MB, fp32)
+  %softmax.88  = f32[64,32,2048]{...} exponential-normalize(%fusion.1423a)
+  %fusion.1423b = bf16[64,32,128]{...} fusion(%softmax.88, %v_cache), kind=kLoop
+      // fusion declined: projected vmem 2,097,152 B > cap 1,572,864 B (cl/7841917)

Consistency check — the arithmetic in one place

S bytes/layer = B·H·s̄·4 B = 64·32·2048·4 = 16,777,216 ≈ 16.8 MB   (write) + 16.8 (read)
Δbytes/step   = 33.6 MB × 32 layers = 1.07 GB
Δt (BW term)  = 1.07 GB ÷ 4.26 TB/s = 252 µs = +6.5% of 3.90 ms
measured      = +320 µs (+8.2%) → residual 68 µs ≈ exposed softmax + lost pipelining
per-layer     : 122 → 132 µs ; ×32 = 3.90 → 4.22 ms ✔ matches alert, traces, and §6 text
v5e tile      : 128·2048·4 = 1.05 MB < 1.5 MB cap → fused ✔ ; v6e: 256·2048·4 = 2.10 MB > cap → split ✔

Artifact 6 — per-op timing diff (the triage first-move, §6.2): op table join, sorted by Δ

op (HLO fingerprint join)              calls   before µs/step   after µs/step      Δ µs        note
────────────────────────────────────────────────────────────────────────────────────────────────────
fusion.1423 (attention_softmax_fused)   32        2,016             —             −2,016      REMOVED
fusion.1423a (qk_scores)                32          —             1,088           +1,088      NEW
fusion.1423b (attn_v)                   32          —             1,184           +1,184      NEW
softmax.88 (exponential-normalize)      32          —               256             +256      NEW (VPU)
fusion.201 (qkv+out+ffn GEMMs)          32        1,888           1,890               +2      unchanged
ppermute.77 (ICI)                       32          128             127               −1      unchanged
──────────────────────────────────────────────────────────────────────────────────────────────
kernel-time sum                                   4,032           4,545             +513
wall step time                                    3,900           4,220             +320

Reading it like an operator: the delta is entirely one op family (1423 → 1423a/b + softmax.88); everything else is noise-level. Kernel-time sum ≠ wall time on both sides — DMA and the VPU softmax partially overlap MXU work — which is why the wall Δ (+320 µs) is less than the kernel Δ (+513 µs); trusting kernel sums over wall clocks is a classic triage mistake, and the table shows why.

Artifact 7 — detector evaluation console (the statistical verdict as it prints)

[gate-eval xla-rc-2026-06-17] family=chat-8b platform=v6e-4
  series tpot_p50/decode.B64.s2048.int8 : base med=3.900 (n=30, MAD=0.017)  rc med=4.220 (n=5)
    mann-whitney U=150 p=6.6e-7 | BH-q(2,160 tests)=1.2e-9? -> q=4.3e-4 PASS
    effect=+8.21% > max(MDE=1.0%, 2sigma=0.9%) PASS  ->  verdict: REGRESSED
  series tpot_p50/decode.B8.s2048.int8  : +7.4%  q=1.1e-3  REGRESSED
  series ttft_p50/prefill.8k            : +0.31% q=0.74    within-noise
  series tpot_p50 @ v5e-8 (all cfgs)    : max |Δ|=0.4%     within-noise
  family verdict: BLOCKING (2 gated series, spend-weighted impact 0.6% fleet)
  -> page: ml-perf-triage | auto-attach: opdiff/00412, bisect job queued

Artifact 8 — executor run-log header (the hygiene that makes n=5 mean anything)

run_id=t1-00412-r3  harness=perfbench-2.31  chipset=v6e-4/cell-mtv5-c12 (pinned set #7, unchanged 41d)
warmup: 120s discarded (compile-cache HIT 14/14 executables; allocator steady after 38s)
load: open-loop poisson λ=matched, trace-mix chat-8b@2026-06 (p50 6k/p95 32k), seed=00412
env: torch_xla 2.11 / xla-rc-2026-06-17 / runtime pinned; host cpuset 0-15; icebox neighbors: none (reserved)
sanity: roofline-predicted TPOT 2.53ms (ideal) / measured 3.91ms -> achieved-BW 0.647 (band 0.60-0.68 OK)

8 · Production Integration and Rollout Safety

Gating policy — severity is dollars × scope, and every threshold pre-agreed:

Finding	Action
≥ 3% on any top-20 series, or ≥ 1% fleet-spend-weighted aggregate	Blocks the release for affected families; compiler team owns unblock (fix, flag, or negotiated waiver with model-team sign-off)
1–3% on gated series	Release proceeds; P1 bug auto-filed with artifacts, must resolve before next RC (no compounding drift)
Tail-model or sub-MDE movement	Tracked in the trend queue; pages nobody
Registered, expected improvement confirmed	Logged as validated win (§9) — closes the loop with the compiler team

Compiler canary: after the gate passes, the new XLA rolls to 1–5% of replicas per model family, interleaved with baseline replicas on identical traffic slices; automated comparison of TPOT/TTFT/goodput on live traffic with sequential monitoring over 24–48 h. Canary catches what benchmarks can't: real traffic mixes, cache behavior, cross-model interference. Rollback: serving binaries are AOT-compiled per XLA release and both versions' artifacts are pre-staged; rollback = flip the binary pointer, minutes, no recompile storm — the compile cache for release N−1 is kept warm precisely for this.

Org mechanics. One perf-triage rotation (staffed by the ML-perf team, not the compiler team) owns first-touch on every page: confirm signal, attach attribution artifacts, route — compiler-owned (HLO diff shows decision change) vs model-owned (retrain/shape change) vs infra-owned (baseline/harness). Routing target: < 1 h, because mis-routed perf bugs are where trust dies. Compiler team owns culprit fix; model teams own model-side mitigations; the detector team owns false-page rate as a KPI reviewed monthly. Trust maintenance is explicit: every page ends with a one-click "true/false regression" verdict from the triager; false-page rate > budget for two consecutive months forces a threshold/noise-floor review. A gate that pages wolf gets ignored the quarter after — alert fatigue is a system-design failure, not an on-call character flaw.

Staff signal: the gate policy separates block vs bug by pre-agreed dollar thresholds (no per-release negotiation); rollback is designed as pre-staged artifacts rather than a recompile; and the triage rotation with routed ownership + a false-page KPI treats organizational trust as a first-class reliability property of the system.

Interviewer follow-ups (§8)
Q: A blocking regression's fix needs 3 weeks, but the release carries a critical security patch. Walk me through the call.
A: Decouple the axes: security patches ride a hotfix lane cherry-picked onto the currently-deployed release, so perf gating never hostages security. If the org truly has one train, the gate's waiver path activates: affected model families pin release N−1 (per-family version pinning is a designed capability, not an emergency hack), everyone else takes N, and the waiver has an owner and an expiry. The design point: the gate must have a narrow, auditable bypass — otherwise the first emergency demolishes it.
Q: Model teams start gaming the gate — tuning their models to the benchmark configs. Is that a failure?
A: Mostly it's the system working — the benchmark configs are production-derived, so optimizing for them is optimizing production. The failure mode is divergence: if teams tune for the suite while production mix drifts, goodput improves on paper only. Defenses: configs regenerate from production traces monthly (moving target), and the canary tier scores live traffic, which can't be gamed. Benchmarks set the gate; canaries audit the benchmark.
Q: Who pays for the benchmark fleet, and how do you defend the budget in planning?
A: Central infra budget, defended with the incident ledger: chip-hours spent vs regression-dollars caught (the §1 math: one blocked 1% fleet regression pays for the year). The detector's own report card — caught/missed/false-page counts and validated-win confirmations — is published quarterly. A perf gate that can't quantify its own ROI becomes the first cost-cut casualty, and then the fleet re-learns why it existed.

9 · Evolution and Hard Problems

10× models (3,000): the per-model paradigm dies; the HLO-fingerprint clustering becomes the primary object — you gate program clusters (a few hundred) plus every model above a spend threshold. Auto-adoption (§3 follow-up) becomes the ingestion path, and per-op diffing must be fully automated triage, since humans can't read 3,000 series.

Pre-silicon qualification: before Ironwood-next tapes out, the suite runs on the architectural simulator / emulation farm. Fidelity limits are respected explicitly: simulators rank kernels and expose gross lowering bugs but miss DVFS, HBM refresh interference, and real collective congestion — so pre-silicon verdicts are "relative regressions vs simulated baseline," never absolute SLO claims, and first-silicon T2 re-baselines everything. The correlation report (simulated Δ vs measured Δ on the previous generation) is maintained as the simulator's own trust metric.

Improvements as first-class: a detector that only frowns trains compiler teams to fear it. Expected-win registration (§4 follow-up) plus validated-win reporting turns the same machinery into the compiler team's scoreboard — and keeps the statistics honest, because an unregistered +4% is investigated with the same energy as −4% (it's often a broken benchmark, occasionally a silent numerics change: faster-but-wrong is the worst outcome the system can bless).

Cross-hardware comparability: "model X is 1.3× faster on B than A" is only meaningful with geometry- and SLO-matched configs: same effective batch per achieved-bandwidth, shapes aligned (or deliberately not) to each MXU/TensorCore tile, same SLO point, stated price basis. The suite's geometry-matched stratum exists so these claims can be made honestly; the labeled production-shaped stratum exists so nobody mistakes a padding artifact for a hardware property. Publishing both numbers with the delta explained is the only defensible cross-vendor statement.

10 · 60-Minute Delivery Run Sheet

Minutes	Beat
0–5	Verbatim opening: "Before architecture — three numbers. The naive matrix is 300 models × 4 platforms × 6 configs × 5 repeats × 8 releases a week ≈ 72,000 chip-hours — a pod doing nothing but benchmarking, so this is a coverage-per-chip-hour design. Second: the detector needs its own SLOs — 24 h verdict, ≤5 false pages/week, 1% detectable effect on the top-20, because 1% of fleet is ~$10M/yr. Third: two customers — compiler team needs a culprit CL, model teams need a trace and a mitigation — so attribution is a product, not a log. Now the architecture."
5–15	Whiteboard order: architecture flow (§2) → tier pyramid with the budget line reconciling to ~6% of naive (§5). State the no-chips-in-pre-submit decision aloud.
15–40	The three deep-dives, steered in this order: benchmark honesty (metric definitions + MFU lies + geometry-matched vs production-shaped), statistics (variance table → noise floors → changepoint + FDR → power formula, n derived), worked attribution example (draw the before/after one-layer timeline; do the 16.8 MB × 2 × 32 ÷ 4.26 TB/s = 252 µs arithmetic on the board; explain v5e/prefill selectivity via the roofline).
40–50	Production integration: gate policy table, canary + pre-staged rollback, triage rotation and the false-page KPI. This is where organizational-maturity signal lives — don't skip it for more statistics.
50–58	Closing beats: improvements-as-first-class (faster-but-wrong is the worst blessing), pre-silicon simulator fidelity honesty, cross-hardware comparability rules. Offer the 10× evolution if time remains.
58–60	Validation close: "The system's own report card — caught, missed, false pages, validated wins — is published quarterly; a gate that can't prove its ROI and its trustworthiness gets ignored, and then it doesn't matter how good the statistics were."

Every number in this document traces to the §6.3/§7 config block or to a stated assumption; anything labeled an assumption (achieved-BW fraction, $/chip-hr, variance magnitudes) should be replaced by fleet measurement before any of it is quoted as fact.