Read as mem_busy on one card; checked against an H200, a B300, a PRO 6000 and a second H100, and against a prediction committed before the run
A decode step on Qwen3-8B keeps one card's memory controller busy
87 % of the time; on gemma-4-26B-A4B, 38 %. Read that
number on one card. It predicts which of the two gains more from a faster card, from a second card,
or from faster memory, and which loses more to slower memory — on hardware it was never measured on. It
does not say by how much, and a prediction committed before the data says so.
Every projection in this repository was built from two Radeons under a desk and whatever Colab offered. The rented time added six machine configurations on the decode and prefill ladder — H100, H200, B300, RTX PRO 6000, two H100s, two PRO 6000s — and eight on the collective. The run produced 586 ladder measurements and 385 collective cells with 0 errors, pushing context past 32 000 tokens for the first time here. Two platform controls come first. An A100 and an L4 rented on the same platform reproduce Colab's committed rows to 0.07 % and 0.9 %, which lets each ratio below be read as a card difference.
The finding is one counter. In campaign-2026-09-02d, the memory controller's
busy fraction on one card (mem_busy) decided what a second Radeon was worth: 90 % and
1.70× on the bf16 8B, 56 % and 1.20× on the w4a16 12B. That was five cells on one box.
The counter now orders the outcome across five settings: one on the pair, and four against an H100 baseline.
Those four are an H200, a B300, a PRO 6000 — the other direction, because its memory is slower — and
a second H100 over NVLink, which is worth 2.9 % to the model that reads least.
It does not predict magnitude. A prediction for the H200 was committed before that run.
The ordering held at both ends, but the spread came back nearly double. At the memory-bound end, the gain
exceeded the memory-clock ratio itself, which the formula used cannot produce. Ordinal, not cardinal,
and on the record as such.
The wire is not the axis, and now that is measured. Across eight configurations the collective's bandwidth end spans 62× and its latency end 3.2×. Batch-1 decode reduces one row and lands on the latency end. NVLink is worth 20 % at two cards and 171 % at four. Three further findings emerged. Four cards chose three attention backends with no flag set. A $7.10/h B300 loses to a $3.95/h H100 on the MoE, but wins by 66 % on the 8B. And past 32 000 tokens, a bounded attention window flattens the decode curve rather than a recurrent state — which falsified a prediction written into the runner before the run.
Every ratio below divides a card rented on Modal by an A100 or an L4 measured on Colab in August. Cross-platform comparisons assume the two environments behave alike. To test that assumption directly, the same two cards were rented on Modal and run on the same models at the context budget and ladder Colab used.
gemma-4-12B decode on an A100-SXM4-80GB at four rungs: Colab on
2026-08-29 and 2026-08-30, Modal on 2026-09-03. The Modal sitting lands inside 0.07 %
of Colab's second on every rung, closer than Colab's own two sittings agree with each other. The band
rests on four Colab rows committed since August. The L4
control says the same on a card that appears in no conclusion, 25.29 tok/s at 32 000 against
Colab's 25.07 and 25.17.
The metric mem_busy_pct records the fraction of samples in which the memory controller
was busy. The same sampler reads it on every machine (harness/telemetry.py: amdgpu's
mem_busy_percent on Radeons, NVML's utilization.memory on CUDA). It represents
a busy fraction rather than achieved bandwidth — the harness marks achieved bandwidth absent on both
platforms and explains why. On 2026-09-02, reading it on one Radeon across two models decided what the
second card was worth. Five settings now test that ordering, holding the model fixed in each and comparing
the two ends of the counter's range:
mem_busy on that base at the same
rung. The pair's own setting is 2026-09-02d and its two models are the bf16 8B and the w4a16 12B,
which is what that campaign ran; the four rented settings compare the two ends of the H100's own
range, Qwen3-8B at 87 % and the MoE at 38 %. In every setting the memory-bound
end moves further from 1.0 than the compute-bound end, in whichever direction the setting moves
— including the PRO 6000, whose memory is slower, where the model that gains most from more
bandwidth loses most from less. Read the last row twice: at 38 % mem_busy a second
H100 over NVLink is worth 2.9 %.
An ordering across five settings is a correlation until it predicts something. An H200 provides the
cleanest test: it matches the H100 in compute capability, 700 W cap, and SM clock ceiling, differing
only in faster memory. If a decode step spends a fraction f of its time waiting on memory and the
rest on everything else, multiplying memory speed by r gives 1 / ((1 − f) + f / r).
Here f was taken as mem_busy on the H100 baseline. The ratio r came from the two
cards' measured maximum memory clocks — 2 619 against 3 201 MHz, so 1.222 — rather
than vendor bandwidth figures unmeasured by this repository. The six predictions were committed before the run.
gemma-4-31B uses 2 000 because its H200 500-rung rounds came back
13 % apart. The two ends are in the right order and clearly separated. The spread is not: measured
0.210 against 0.114 predicted, and Qwen3-8B's 1.254×
exceeds r itself, which that form cannot produce for any f ≤ 1, so the error
is in f and not in r, and not by a constant scaling. The counter orders and does not
size. An independent check that the
mechanism is right where the arithmetic is not: on the H200 mem_busy is lower than on the
H100 for every one of the six models, with no crossings — faster memory doing the same work leaves
the controller busy for less of the time, which the model says should happen and was not fitted to.
The a100 article's section 9 used to say the second Radeon's bandwidth was spent on the interconnect.
Campaign allreduce-2026-09-02 withdrew that claim with direct measurements on the pair.
Under graph replay, a collective took 16.6–21.5 µs — roughly a tenth of a step — paid
equally by a model the second card helps 1.70× and one it helps 1.20×. The same sweep across seven
NVIDIA configurations gives the withdrawal a mechanism.
t_graph_us at hidden 4 096 — Qwen3-8B's row
— for one token, which is a decode step, and for 16 384 tokens, which is a prefill; NCCL
2.29.7 on every NVIDIA run, RCCL 2.27.7 on the Radeons, the same script. The bandwidth end spans
62× across the eight and the latency end 3.2×; among the
pairs alone the latency end spans 1.5× against the same 62×. Batch-1 decode
reduces one row and lands on the latency end, where a B300 pair beats the Radeon pair by 1.2×
while beating it by 62× on the other end. That is why the second card's value tracks the memory
controller and not the interconnect: the interconnect's range is real and inference never enters it.
What NVLink is for is the fourth card. Adding cards three and four costs the H100
×1.22 on a decode step and ×1.30 at the bandwidth end; the PRO 6000, on PCIe with no NVLink,
×2.71 and ×2.99. Two cards without NVLink cost 20 % over two with it; four cost
171 %. And four H200s agree with four H100s to 1 % at both ends, so none of the H200's ladder
advantage in section 3 arrives through the wire.
One vLLM, one command line, no backend flag anywhere in the sweep. The runner reads the backend
directly out of every serve log rather than trusting a table, which is the only reason this split is
known. Because vLLM 0.28 has no flag for the decoder's backend (attention_backend exists only
in config/speculative.py, read out of the installed package), the control that would separate
the card from the kernel is not available.
Qwen3-8B, by 4 % on gemma-4-12B, and
loses on gemma-4-26B-A4B — the same ordering as the counter's, because the
B300's advantage is bandwidth and the MoE reads least. Per dollar it loses on all three. The B300 also
changed attention backend, so machine and backend move together in its column; the prices are Modal's
list on 2026-09-03 and are the only typed numbers on this page.
Every ladder in this repository stopped at 32 000 tokens until this sweep, where the H100 carried
five models to 128 000. A prediction was written into the runner before the run. The models featured
three distinct attention designs: gemma-4 reads a KV cache that grows without bound; Qwen3.8-27B
carries a recurrent state that does not grow at all; and Muse-Glimmer-30B attends through a
2 048-token window that grows and then stops. The prediction placed Muse-Glimmer between the two.
Muse-Glimmer-30B keeps all but 4.8 % of its
500-token rate at 128 000; the hybrid SSM loses 21.8 %, as far as the dense
31B's 22.0 %. The recurrent-state model is not the flat one. What flattens a curve is a bounded
attention window, not a constant-size state — the SSM still carries attention layers whose cost
grows with depth. The same ordering appears in an independent quantity: moving from an A100 to this
H100 at 32 000 is worth 1.36× to Muse-Glimmer, 1.45× to the SSM and
1.90–2.16× to the three gemma models. One confound, and it runs against the conclusion:
Muse-Glimmer is the one model whose W4A16 kernel changes between machines, Machete on the Hoppers and
Marlin elsewhere, and Machete is the faster of the two on Hopper.
Section 3 has the limit. The form 1 / ((1 − f) + f / r), with f read off
one card, under-predicts the spread by half and cannot produce the memory-bound end at all. Either the
effective bandwidth ratio exceeds the memory-clock ratio — the vendors' figures give 1.43, at which the
same form over-predicts every row — or the H200 differs from the H100 in something the prediction
assumed away. Nothing here separates those.
Figure 6's B300 column and every gemma-4 comparison conflate the card with its backend. Because vLLM 0.28 cannot fix the decoder backend, this was not measured.
gemma-4-26B-A4B cannot run at TP=4: CompressedTensors WNA16 MoE with static group
scales requires the MoE intermediate size per tensor-parallel partition to be divisible by
group_size (32) — a property of the checkpoint, not a failure to retry. The other four models at
TP=4, and Muse-Glimmer at TP=2 on the rented pairs, were not bought.
mem_busy is a busy fraction: 87 % at one rate is not 87 % at another, and achieved
bandwidth is absent on both platforms. The counter travels; its denominator does not. It also spans two
driver metrics: amdgpu's mem_busy_percent on Radeons and NVML's utilization.memory
on CUDA. That the same two models land at 90/56 on one and 87/53 on the other is evidence the
two counters measure alike. It is not proof.
The H200, B300, PRO 6000, and second-H100 rows all divide by one H100 sitting, so any error there shifts four rows together. Platform controls bound that sitting against Colab to 0.07 %, and the pair ran on separate hardware. The five are one independent check on the pair's finding, not five.
Twenty-three prefill cells failed the repeatability cut on the rented cards, all of them between
500 and 2 000 tokens, on the cold engine's first requests. Two decode cells failed it too, both on
the H200: gemma-4-26B-A4B at 1 000 and gemma-4-31B at 500. Nothing above
depends on them.
Everything here. The campaign directories, runners, serve logs, and collective
sweep are in
benchmarks/cuda-modal/.
The prediction and appended verdict are in
cuda-h200/campaign-2026-09-03/PREDICTION.md.
Every number here is recomputed from committed rows by
verify_doc_figures.py,
except two marked list prices.
The counter itself, and the five cells on the pair that made it worth testing, are a different article's section 9.