Staff-level model answer · 60-minute system design round · TPU-primary (v5e / Trillium / Ironwood) + GPU fleet (H100 / B200) · July 2026
Hardware price anchors: GCP public list (Ironwood $12.00 OD / $5.40 3-yr CUD per chip-hr; Trillium $2.70 / $1.22; v5e $1.20 / $0.54) and mid-2026 GPU market medians (H100 ≈ $2.3–3.1/hr; B200 ≈ $5.9/hr). Internal amortized rates assumed below.
The first move in the room is to establish that "routing" here is a joint optimization at three timescales, not a load balancer. Which model variant serves a request, which hardware pool it lands in, and which replica within the pool are coupled decisions: the variant choice constrains which pools are feasible (no native FP8 below Ironwood on the TPU side), the pool choice determines the marginal cost of the tokens, and the replica choice determines whether a cached prefix is reused or recomputed. But they operate at different timescales, and conflating them is how designs collapse. I decompose explicitly:
| Question | Assumed answer |
|---|---|
| Traffic scale and shape? | ~1B requests/day, peak 50K req/s aggregate; mean 2K prompt / 300 output tokens → peak ~100M prefill tok/s, ~15M decode tok/s fleet-wide. ~60% interactive, 40% batch-tier by token volume. |
| SLO classes? | Three classes. A (agentic/interactive-fast): TTFT p99 ≤ 500 ms, TPOT p99 ≤ 30 ms. B (standard interactive): TTFT p99 ≤ 2 s, TPOT p99 ≤ 80 ms. C (batch): deadline-scheduled (e.g., complete within 24 h), no TTFT bound. |
| Quality requirements? | Per-product minimum quality bars measured on per-product eval suites. Some products permit quality-for-cost trades (routing to distilled/quantized variants); some pin an exact variant. The product team owns the bar; the platform owns the policy that respects it. |
| What is "cost", precisely? | Chip-seconds × amortized internal $/chip-hr → $/token, attributed per request. I'll assume internal amortized rates near 3-yr committed pricing plus overheads: v5e $0.80, Trillium $1.80, Ironwood $6.00, H100 $2.40, B200 $5.00 per chip/GPU-hour (assumptions, flagged in §2). |
| Multi-region? | Yes, but I'll design a single region and note that cross-region routing adds a slower control loop (spillover on capacity exhaustion, ~100 ms RTT penalty) rather than changing the architecture. |
| Does prefix locality matter? | Strongly: ~55% of interactive tokens share long system-prompt/tool-schema prefixes (multi-turn chat and agent loops), so cache-aware replica selection is a first-class cost lever, consistent with what Dynamo/llm-d-class routers exploit in production. |
| Do we own the models? | Yes — we can distill, quantize, and create variants. (Follow-up in §8 covers the third-party case.) |
Functional: route every request to a (variant, pool, replica) satisfying its SLO class and quality bar at minimum expected marginal cost; enforce per-tenant admission; support cascade escalation; expose degradation ("brownout") as an explicit, product-visible policy; attribute cost per request/tenant.
Non-functional, for the router itself: routing decision adds ≤ 1 ms p99 to the request path; router availability strictly above any single serving pool (it must survive pool failures it's routing around, so: stateless decision path, replicated soft state, static fallback tables); decisions must be explainable post-hoc (log the full decision vector per request); safe under predictor failure — every learned input has a dumb fallback.
Before drawing any boxes, I build the cost surface the router will optimize over. Every routing decision is ultimately a comparison of marginal $/token across feasible (variant, hardware, config) cells, so if the arithmetic here is wrong, the router is confidently wrong at 50K req/s. The model separates prefill (compute-bound: FLOPs against peak matmul throughput × achievable MFU) from decode (bandwidth-bound: bytes of weights + KV read per step against HBM bandwidth × achievable utilization, amortized over batch occupancy).
prefill $/tok = (2P FLOP/tok) / (chips · peak_flops · MFU) · (chips · $/chip-s) decode $/tok = step_time / B · (chips · $/chip-s), step_time = (W_bytes + B · ctx · kv_bytes/tok) / (chips · HBM_BW · util)
Two consequences fall out immediately and shape the whole design. First, decode $/token is roughly invariant to tensor-parallel width: doubling chips doubles both aggregate bandwidth and cost, so TP width is a latency (TPOT) knob, not a cost knob — the router uses config choice to buy SLO headroom, not savings. Second, weights are read once per step regardless of batch, so $/token falls almost hyperbolically with batch occupancy until KV traffic or compute dominates. Occupancy is the strongest cost lever in the system.
| Variant / config | Trillium v6e | Ironwood | H100 | B200 |
|---|---|---|---|---|
| 70B BF16 prefill / decode $/Mtok · TPOT |
8-chip $0.139 / $1.34 21.5 ms |
2-chip $0.184 / $0.995 19.1 ms |
TP8 $0.189 / $0.949 11.4 ms |
TP2 $0.173 / $0.827 19.1 ms |
| 70B FP8/W8 INT8 on v6e (no native FP8) |
4-chip $0.069 / $0.671 21.5 ms |
1-chip $0.092 / $0.498 19.1 ms |
TP2 $0.094 / $0.474 22.8 ms |
TP1 $0.086 / $0.413 19.1 ms |
| 8B distilled FP8/W8 | 1-chip $0.0079 / $0.122 15.6 ms |
1-chip $0.0105 / $0.090 3.5 ms |
1 GPU $0.011 / $0.086 8.3 ms |
1 GPU $0.0198 / $0.075 3.5 ms |
Note the 8B × Ironwood cell: at B=64 the expensive chip loses to H100/B200 — 8 GB of weights and 8.6 GB of KV against 192 GB of HBM is capacity thrown away. Pushed to B=512 (68 GB KV) the same cell reaches $0.052/Mtok and wins the row. Occupancy again.
Decode bytes/step = weights 70 GB + KV 64·2048·160 KB = 70 + 21.5 = 91.5 GB.
Effective BW = 7.37 TB/s · 0.65 = 4.79 TB/s → step = 91.5/4790 = 19.1 ms → 64 tok / 19.1 ms = 3,350 tok/s.
Chip $ = $6.00/hr = $1.67e-3/s → decode = 1.67e-3 / 3,350 = $0.498/Mtok.
Prefill: eff FLOPs = 4,614 · 0.55 = 2,538 TFLOPs → 2.538e15 / 140e9 = 18.1K tok/s → 1.67e-3 / 18.1e3 = $0.092/Mtok.
Compute-bound sanity check for decode: 64 · 140 GFLOP = 9.0 TFLOP/step → 9.0/2,538 = 3.5 ms of compute vs 19.1 ms of memory time → bandwidth-bound, model valid. (At B = 256: 14.1 ms compute vs 32.6 ms memory — still BW-bound. In fact at ctx 2K with FP8 KV, decode never goes compute-bound here: marginal HBM per token-step is 2048 · 160 KB / 4.79 TB/s = 68 µs vs. 55 µs of marginal compute, so $/tok asymptotes to the KV-traffic floor of ~$0.11/Mtok rather than crossing a compute roofline. The crossover exists only at shorter contexts.)
B = 16: bytes 70 + 5.4 = 75.4 GB → 15.7 ms → 1,017 tok/s → $1.64/Mtok
B = 64: 91.5 GB → 19.1 ms → 3,350 tok/s → $0.498/Mtok
B = 256: 70 + 86 = 156 GB → 32.6 ms → 7,850 tok/s → $0.213/Mtok (TPOT 32.6 ms — fails class A, fine for B/C)
An 8× cost range on identical silicon from occupancy alone — wider than the spread between most hardware columns. This is the number I put on the whiteboard and keep pointing back at.
(i) Batching density is a cost lever as strong as model choice. Moving a request from 70B BF16 to 70B FP8 saves ~2×; moving a decode pool from B=16 to B=256 saves ~8×. So routing that consolidates traffic — fewer, fuller pools; batch-tier work admitted specifically to fill occupancy troughs; anti-fragmentation in replica selection — buys more than per-request cleverness. A router that scatters load "fairly" across half-empty replicas is actively destroying money while every dashboard shows green.
(ii) Hardware-generation arbitrage is real but second-order, and it's about granularity and sunk fleet, not raw $/token. At my assumed rates, 8B on already-owned v5e (2-chip, INT8: bytes 16.6 GB / (2·0.82·0.65 = 1.07 TB/s) = 15.6 ms → 4,100 tok/s at $1.60/hr → $0.108/Mtok) is beaten by B200 ($0.075) on paper — but the v5e fleet is depreciated, single-chip-granular (no TP fragmentation), and its marginal cost approaches power. Latency-tolerant class-C 8B traffic goes there precisely so Ironwood/B200 capacity stays reserved for the traffic that needs it. The arbitrage is an opportunity-cost argument, and the router is the mechanism that executes it.
"Your MFU/BW-utilization numbers are assumptions. What if they're off by 20%?" The absolute $/Mtok shifts but the lever ordering doesn't, and the design only depends on the ordering plus continuously-measured cell throughputs (§5) — the static table is a prior, not the operating truth. The one place absolute error bites is fleet planning, which is why the planner consumes measured, not modeled, $/token.
"Why amortized internal rates instead of on-demand prices?" Because the fleet is owned/committed capacity: the marginal cost of routing a request is the opportunity cost of the chip-seconds, not a rental price. This changes decisions — e.g., filling occupancy troughs with batch work is nearly free on committed capacity, whereas on-demand pricing would (wrongly) tell you to scale to zero.
"Prefill is 5–10× cheaper per token than decode in your table. Does that surprise you?" No — it's the arithmetic-intensity gap: prefill does 2P FLOPs/token against compute; decode moves all weights per step against bandwidth. It's why disaggregation (§4) works, why prefix caching pays twice (skips compute but the bigger win is TTFT), and why output-heavy workloads dominate cost attribution.
Variant selection is where cost and quality actually trade against each other, so it gets the most explicit governance. The policy space per tenant is a pre-approved variant ladder (e.g., 8B-distilled → 70B-FP8 → 70B-BF16) with a per-product quality bar deciding which rungs are reachable and under what conditions.
Upfront routing: a cheap difficulty estimator scores the request before any LLM runs, and the router commits to a variant once. The estimator is a small classifier (a few-hundred-M-param encoder or gradient-boosted model over features: prompt length, tenant, task type, retrieval hit count, historical per-tenant difficulty distribution, embedding-space distance to known-hard clusters). Latency ~1–2 ms on host CPU or a sliver of accelerator — inside the router budget. Its failure mode is silent misclassification, so it is calibrated (§5) and biased conservative: uncertain → bigger model.
Cascade: run the small model, escalate on low confidence (mean token logprob below a calibrated threshold, self-assessment token, or a verifier head). Cascades see the actual attempt, so they misroute less — but the escalated fraction pays both models and an extra serial round-trip.
All-70B-FP8: 2048 · $0.086/M + 300 · $0.413/M = $1.77e-4 + $1.24e-4 = $3.0e-4/req
All-8B: 2048 · $0.0198/M + 300 · $0.075/M = $4.1e-5 + $2.3e-5 = $6.3e-5/req (4.8× cheaper)
Cascade, 25% escalation (escalatees burn full 8B prefill + ~32 assessment tokens, then full 70B pass):
= $6.3e-5·0.75 + 0.25·($4.1e-5 + 32·$0.075/M + $3.0e-4) ≈ 4.7e-5 + 8.6e-5 = $1.3e-4/req → 2.3× cheaper than all-70B.
Escalation latency tax on the 25%: 8B prefill 29 ms + 32 tok · 3.5 ms + 70B re-prefill 127 ms ≈ +270 ms TTFT (prefix-cache transfer of the 8B KV doesn't help — different model, incompatible cache).
Decision rule: cascade wins where escalation rate is low and the SLO class can absorb +270 ms on the tail (class B/C). Class A gets upfront routing only. Break-even escalation rate vs upfront routing ≈ where cascade cost + misroute cost curves cross; with these numbers, cascades stop paying above ~65–70% escalation.
Quality bars are meaningless unless measured the way they're enforced. Per (product, variant): offline eval suites owned by the product team, evaluated per (variant, hardware-numerics) pair — W8A8-INT8 on Trillium and FP8 on Ironwood are not the same model even when the checkpoint is, so the same ladder rung has hardware-dependent quality — and re-run on every variant change (new quantization, new distill, new compiler release: quantization regressions are numerically silent and workload-dependent); online, a sampled shadow stream (~0.1–1%) scored by LLM-judge with periodic human audit, sliced per traffic class, never aggregate-only (Simpson's-paradox pathology, §7). The contract: product owns the bar and the eval; platform owns the routing policy and must prove, per policy change, that the bar holds. Any brownout rung below the bar requires product sign-off and is surfaced in the response metadata — degradation is a product-visible state, never a silent trade.
A cascade's cost win is conditional on the small model staying good for the current traffic distribution. Escalation rate per (tenant, task-type) is therefore a first-class SLI with alerting on drift (CUSUM/EWMA against baseline): rising escalation means the cost win is evaporating in real time; falling escalation with flat quality scores can mean the confidence signal broke and garbage is passing — both directions page. This is the difference between a cascade and a slow-motion quality incident.
At admission we know: tenant, SLO class, prompt tokens (exact), tool/system-prompt hash → prefix-cache key (block-hashed, radix-prefix style), and variant eligibility from §3. We do not know output length, and almost every downstream estimate — decode-slot occupancy, completion time, KV memory reservation — depends on it. So we predict it: a small regressor over (tenant, task type, prompt features, stop conditions, historical per-tenant output distribution), predicting a quantile band, not a point (reserve at p70, admission-check at p95). Misprediction consequences are asymmetric: under-prediction over-commits KV memory → evictions/preemption storms mid-decode (expensive: evicting a 2K-context sequence discards ~0.3 GB of KV state and forces recompute); over-prediction strands capacity. Calibration by tenant, monitored in §5, with a conservative static fallback (per-tenant p95 historical).
Replica choice inside a pool is a two-term score, in the same shape Dynamo's KV-aware router and llm-d's precise (KV-events-fed) scorer converged on: score = α · prefix_overlap_benefit − predicted_queue_penalty, over a router-side radix/prefix index maintained from engine cache events. The point of the arithmetic is to set α from first principles instead of vibes:
Lost 4K-token prefix → recompute: 4,096 tok / 16.1K tok/s = 254 ms added TTFT; cost 4,096 · $0.086/M = $3.5e-4 (≈ the entire serving cost of an average request, burned once).
KV bytes at stake: 4,096 · 160 KB = 655 MB.
→ Route to the cached replica unless its predicted queueing delay exceeds the alternative's by > 254 ms (latency view); the cost view favors the cached replica even past that point, so class C tolerates far more queue skew for cache affinity than class A. α is therefore per-SLO-class, and the break-even is recomputed from live prefill throughput, not hard-coded.
Queue depth in requests is a lagging, lying signal: ten 100-token requests ≠ one 32K-token request. The router tracks predicted work: outstanding prefill tokens (compute-seconds) and decode-slot occupancy (predicted via output-length bands), separately, normalized per replica by that replica's measured throughput — heterogeneous cells make raw counts meaningless.
Algorithm choice, justified at scale. A single global optimal scheduler at 50K req/s with a ≤1 ms budget is out; a fully random/round-robin layer wastes the cost model. I commit to a hierarchy: (1) the global policy layer resolves (variant, pool) from pushed routing tables (updated ~10 s) — a table lookup plus a cheap cost-normalized comparison across 2–3 candidate pools; (2) the cell-local scheduler (one per pool, ≤ low-thousands of replicas, full soft state: radix index + work estimates) picks the replica with weighted power-of-two-choices over cost-normalized predicted completion time: sample two candidates biased by prefix overlap, score both with the α-rule, take the winner. P2C keeps herding down under stale state (its classic property), makes the decision O(1), and degrades gracefully to plain P2C-by-work when the prefix index is unavailable. On TPU cells the scheduler is additionally bucket-aware: XLA static shapes mean prompts are padded into compiled length buckets, so the scheduler packs admissions toward bucket boundaries — padding waste is a router-visible cost term, a consideration GPU-side continuous batching doesn't have.
Prefill and decode are different machines economically (§2), so class-A/B traffic runs disaggregated (Splitwise/Mooncake-lineage, as productized in Dynamo-style stacks): the router makes two placement decisions with a KV-transfer edge between them. Prefill pool choice optimizes compute cost + cache reuse; decode pool choice optimizes slot occupancy + TPOT headroom; the transfer cost gates which pairs are viable.
NVLink domain (assume ~350 GB/s effective): 328/350,000 ≈ 0.9 ms — free; pair freely within a domain.
Ironwood ICI cross-slice (9.6 Tb/s peak per chip; assume ~200 GB/s effective end-to-end): ≈ 1.6 ms — cheap; ICI domain is the pairing boundary on the TPU side.
Cross-domain DCN at 200 Gbps NIC (~15 GB/s effective): ≈ 22 ms — tolerable for class B, and it overlaps with decode-queue wait; but at 32K context (5.2 GB) it becomes 350 ms → cross-DCN pairing is banned for long-context class A. The router carries a per-(prefill-cell, decode-cell) transfer-cost matrix and treats it as one more edge weight.
Per-(tenant, SLO-class) token buckets on predicted work (prefill-compute-seconds and decode-slot-seconds, not request counts). Class C sits in a deadline-scheduled queue (EDF within cost-tier) and is the shock absorber: it back-fills occupancy troughs and is preempted first. Overload runs a staged, product-visible brownout ladder: (1) defer class C; (2) shift eligible class-B traffic down one variant rung where the product pre-approved it, tagged in response metadata; (3) shed class B by tenant priority with explicit 429/retry-after; class A sheds last. Every stage is a declared state machine transition with hysteresis — never an emergent behavior of a scoring function.
"Why P2C at the cell instead of a central optimal assignment?" At cell scale a central scheduler is actually feasible — and on TPU cells I'd lean further central because deterministic execution + static shapes make completion-time prediction tight enough to trust. The hierarchy is the real answer: the global tier must be table-driven for the latency budget and blast-radius isolation; within a cell, P2C-with-scoring gives ~all of central's benefit with none of its single-threaded hot loop, and it's robust to stale-state herding — the balanced-allocations results under stale information back this — where a greedy central scorer is not.
"Your output-length predictor is wrong 30% of the time. Does the system fall over?" No, by construction: reservations use quantile bands with an admission check at p95, decode pools hold an eviction-headroom reserve (~10% of KV memory), and sustained per-tenant misprediction trips the fallback to historical p95 — which costs stranded capacity, not correctness. The pathology table (§7) covers the storm case.
"Doesn't cache-affinity routing fight batching density?" They align more than they fight: affinity concentrates related traffic, which raises occupancy on fewer replicas. The genuine conflict is hot-prefix herding past a replica's capacity; the fix is bounded affinity — replicate the hot prefix's KV to a second replica when its queue-penalty term persistently exceeds the overlap benefit (exactly the imbalance signal the P2C score already computes).
Everything "smart" in this router is an estimate: the cost table, the output-length predictor, the difficulty classifier, the queue/work model, the transfer-cost matrix. The feedback plane is what keeps those estimates honest and the system accountable — and it's where routers quietly rot when it's an afterthought.
The §2 table is a prior. Production continuously measures, per (variant, hardware, config, batch-band, context-band) cell: achieved tok/s, achieved MFU/BW-utilization, occupancy distribution, and therefore measured $/Mtok. The routing tables consume measured values with the analytical model as sanity bound (alert when they diverge > 15%: either the fleet regressed or the model is stale). This matters because the table genuinely goes stale: a single XLA or serving-stack release that changes decode throughput 15% silently re-orders pool preferences fleet-wide. Continuous micro-benchmarks (canary replicas replaying a fixed trace per release) catch this at rollout, not in the monthly cost review.
Output-length, difficulty, and completion-time predictors are trained on production logs (features frozen at admission time to avoid leakage), retrained on a cadence, and monitored for calibration (reliability diagrams per tenant; PSI on input feature drift). Non-negotiable properties for every learned component: a dumb, safe fallback (historical per-tenant quantiles; conservative "route big" for difficulty; plain P2C-by-work for placement), a kill switch (per-component flag flippable without deploy), and decision logging — every request's routing record carries the predictor versions, inputs, scores, and the counterfactual runner-up choice. At 3am, an on-call must be able to answer "why did this request go to that pool" from one log line, and "make the router boring again" with one flag.
No routing-policy change ships on live traffic first. The pipeline: (1) replay — deterministic simulator consumes sampled production traces (arrival times, token counts, prefix structure, measured per-cell service curves) and evaluates candidate policy vs. incumbent; (2) shadow — new policy computes (and logs) decisions on live traffic without acting; (3) staged rollout by cell with automatic rollback on SLO/cost guardrails. The counterfactual metric is defined up front: Δ$ per Mtok at equal-or-better SLO attainment per class, over the same trace — cost deltas quoted without pinning SLO attainment are how routing teams flatter themselves.
"Your replay simulator will diverge from reality. How much fidelity do you need?" Enough to rank policies, not to predict absolutes — the promotion decision is a paired comparison on identical traces, which cancels most modeling error. The known fidelity gaps (feedback effects: routing changes alter cache hit structure, which alters service times) are exactly why shadow mode and staged rollout follow replay rather than replacing it.
"What breaks first if telemetry lags by five minutes?" Nothing on the fast path — per-request decisions run on cell-local state (sub-second) plus routing tables (10 s). Five-minute lag hits pool control: autoscaling and brownout transitions get sluggish, which the hysteresis margins are sized to absorb. The design rule: each timescale layer only depends on telemetry at least one order of magnitude fresher than its own actuation period.
"How do you keep the difficulty classifier from being gamed by tenants who learn that 'hard-looking' prompts get the big model?" Difficulty affects which rung of the tenant's own pre-approved, billed ladder serves them — tenants pay attributed cost, so gaming buys them a bigger bill, not free quality. Where quality is platform-subsidized, the classifier's per-tenant calibration drift monitor is the detection mechanism, and the response is contractual (quota/pricing), not adversarial ML.
Class-B interactive request: 2,048-token prompt of which 1,600 tokens are a shared system-prompt + tool-schema prefix; predicted output 300 (p70 band 180–520); tenant quality bar requires 70B-class; tenant pre-approved FP8.
IW-3: prefill only 448 new tokens: 448 / 18.1K tok/s = 25 ms → TTFT ≈ 120 + 25 = 145 ms; prefill cost 448 · $0.092/M = $4.1e-5.
B2-7: full 2,048-token prefill: 2,048 / 16.1K = 127 ms → TTFT ≈ 20 + 127 = 147 ms; prefill cost 2,048 · $0.086/M = $1.76e-4.
Latency ≈ tie (both ≪ 2 s budget); cost favors IW-3 by 4.3× → route IW-3. Note the structure: the cache hit converted a 6-chip-generation pricing gap into a win for the more expensive chip — prefix locality dominates hardware choice at these overlap lengths.
| Pathology | Symptom | Detection signal | Mitigation |
|---|---|---|---|
| Output-length underprediction | KV memory over-commit → mid-decode evictions → preemption storms; TPOT p99 spikes as evicted sequences recompute | Eviction rate per cell; predictor p95-coverage below target, per tenant | Quantile-band reservation with p95 admission check; ~10% KV headroom reserve; per-tenant fallback to historical p95 on coverage breach |
| Cascade escalation storm | Distribution shift makes the 8B model unconfident fleet-wide → 70B pools absorb 2× load with an extra serial hop; class-B TTFT collapse | Escalation-rate EWMA per (tenant, task) with drift alert; 70B pool admission-queue growth correlated with escalation events | Escalation-rate circuit breaker: above threshold, flip affected traffic to upfront routing at the big variant (skip the doomed 8B attempt); pre-provisioned surge headroom in 70B pools sized to the breaker threshold |
| Cache-affinity herding | One hot shared prefix concentrates traffic on a replica past capacity; queueing eats the cache win | Per-replica work skew vs. cell mean; affinity-score wins with negative realized latency delta (the router grades its own decisions) | Bounded affinity: replicate hot-prefix KV to a second replica when queue-penalty persistently exceeds overlap benefit; P2C's load term provides natural damping |
| Cost-model staleness after a compiler release | XLA/serving release changes decode throughput 15%; routing tables now prefer the wrong pools; fleet cost drifts up with no alert | Release-canary fixed-trace microbench delta; measured-vs-model $/Mtok divergence > 15% | Cost table consumes measured throughput; release gate blocks table promotion until canaries pass; model divergence pages the perf team |
| SLO-class starvation | Class C deadline misses pile up because interactive traffic permanently claims all occupancy backfill | Deadline-attainment SLI per class; class-C queue age p99 | EDF with aging within class C; minimum capacity floor per class in pool control (starvation is a capacity-plan bug, not a scheduler bug — surface it there) |
| Pool oscillation (control instability) | Routing tables flip traffic between two near-tied pools each update; batch occupancy thrashes on both; cost rises while each snapshot looks fine | Table-diff churn metric; per-pool occupancy variance at the table-update frequency | Hysteresis band on pool preference (switch only on > X% sustained advantage); damped table updates (EWMA of cost estimates); randomized per-frontend update jitter |
| Quality regression hidden by aggregate metrics | Aggregate judge scores flat while one tenant's traffic silently degraded (mix shift masks it — Simpson's paradox) | Quality SLIs sliced per (tenant, class, variant), never aggregate-only; mix-shift-adjusted control charts | Per-slice quality bars with per-slice alerting; routing changes gated on worst-slice delta, not mean delta |
| Degraded hardware silently absorbing traffic | A replica with throttled HBM (thermal, ECC masking) runs at 60% throughput; work-based router keeps feeding it "because its queue drains"; its $/Mtok is 1.7× fleet | Per-replica measured $/Mtok vs. cell distribution (outlier detection); throughput-normalization factor drift | Replica-level cost outliers auto-drain to canary status; normalization factors recomputed from measured, not nameplate, throughput |
Because the cost table says the variant lever is ~5× while the hardware lever is 1.6× — collapsing variants throws away the big lever to simplify the small one. The honest part of the challenge is operational: cascades add predictors, drift monitoring, and an org contract, and if a fleet lacks the maturity to run those (§5), a single-variant fleet with excellent occupancy shaping captures the majority of achievable savings at a fraction of the complexity. I'd stage it exactly that way: occupancy + placement first (no quality risk), variants second, cascades last — each stage justified by its own replay-measured Δ$.
The quality-aware layer shrinks and the contracts grow. No distillation, no custom quantization without provider terms permitting it, and quality bars must be validated per provider-pushed model update you don't control — so the eval/canary plane becomes the front line, and variant ladders become across-provider ladders governed by commercial terms (rate limits, per-token pricing replacing chip-second cost as the objective). Placement optimization survives intact for self-hosted third-party weights; for API-backed models the "pool" abstraction becomes a priced external endpoint and the router's cost model consumes list prices instead of §2 arithmetic — structurally the same optimizer with a different cost column.
Four things. (1) Static-shape buckets: XLA compilation quantizes prompt lengths into buckets, so admission batching must pack toward bucket boundaries and padding waste is a router-visible cost term; GPU continuous batching has no analog. (2) Deterministic execution: TPU step times are tight distributions, so completion-time prediction is trustworthy enough to lean more central/greedy in cell scheduling, where GPU jitter demands P2C's robustness. (3) ICI-domain topology: the placement atom is a slice within an ICI domain, and the disaggregated pairing matrix is ICI-domain-aware (cheap intra-domain KV transfer, expensive DCN); on the GPU side the analogous boundary is the NVLink domain. (4) Feasibility matrix: no native FP8 below Ironwood means the variant×hardware table has holes (INT8 stand-ins on v5e/v6e with their own eval'd quality deltas) — variant availability is hardware-conditional, which pure-GPU designs never confront.
This is a counterfactual-accounting problem, and hand-waving it is how routing teams lose credibility with finance. Three tiers of evidence: (1) replay — incumbent vs. baseline policy on identical traces, Δ$ at equal SLO attainment (the clean but simulated number); (2) per-request runner-up logging — every routing record carries the evaluated alternative and its estimated cost, so realized savings are integrable directly from production logs (estimated counterfactual, real traffic); (3) holdback cells — a small fraction of traffic permanently on the naive policy as a live control, the only estimate that captures feedback effects like cache-structure shifts. The honest headline number is (3), with (1) and (2) as attribution detail; and the denominator is always $/Mtok at fixed SLO attainment, because raw $ savings can be manufactured by quietly missing SLOs.
Cost model before architecture — and the lever ranking (occupancy 8× ≥ variant ~5× ≫ hardware 1.6×) derived with shown arithmetic, then used to allocate design effort. Batching density as the real lever — the router framed as an occupancy-shaping machine, with anti-fragmentation and batch-tier backfill as first-class mechanisms. Timescale decomposition — the ≤1 ms path executes pre-computed policy; optimization lives in slower loops with hysteresis. Learned components with safe fallbacks — kill switches, decision logging with counterfactuals, calibration monitoring; smart and accountable as one requirement. Explicit, product-visible degradation — brownout as a pre-negotiated state machine, quality bars owned by products and enforced per slice. The counterfactual savings proof — holdback cells plus runner-up logging, denominated at fixed SLO attainment.
"Before I draw anything, two framing moves. First, 'routing' here is three coupled decisions — which model variant, which hardware pool, which replica — and they run at three timescales: per-request in under a millisecond, pool sizing in minutes, hardware mix in weeks. I'll design the fast path to execute pre-computed policy and put the actual optimization in the slower loops. Second, I want to build the cost model before the architecture, because every routing decision is a $/token comparison and I'd rather we agree on the arithmetic than argue about boxes. Let me confirm scale and SLOs — I'll assume ~1B requests/day peaking at 50K/s, three SLO classes, per-product quality bars, and that cost means chip-seconds at internal amortized rates. The router itself gets an SLO: ≤1 ms p99 added latency and availability above any pool it routes to." Then straight into the prefill/decode cost formulas and the 3×4 table.
(1) The cost table and occupancy sweep — it demonstrates hardware-level fluency and sets up every later argument. (2) Cache-aware routing arithmetic — the break-even derivation shows judgment under a real tension rather than a memorized pattern. (3) The feedback plane — fallbacks, kill switches, replay validation — because it's where most candidates' "smart router" answers have nothing.
Validation methodology (holdbacks + counterfactual logging, Δ$ at equal SLO attainment) as the proof-of-value story, then the TPU/GPU contrast (buckets, determinism, ICI domains, FP8 feasibility) to land the platform-specific expertise.
All $/token figures derive from the stated assumptions (chip rates, MFU/BW utilization, B=64 / ctx 2K reference point) and are internally consistent to rounding; they are priors for a measured-telemetry system, not market quotes. Pricing anchors: Google Cloud TPU list pricing and mid-2026 GPU market medians as of July 2026.