Decode rate and prefill scaling on dual Radeons; checked against a single card, an A100 and an L4
Adding a second consumer card to a batch-1 decode buys 1.11× to 1.23× and no more. At prefill the same card cuts a 32 K prompt from 40.1 s to 21.5 s — 1.87×. Both measurements ran on the same two cards on the same day. What changed is the job they were asked to do.
This article used to ask a single question — two consumer cards against one A100 — and that question bundles three separate issues. Measuring each card on its own pulls them apart: what one consumer card is worth against one datacentre card, what a second consumer card adds, and only then what the pair delivers.
The second card adds little at decode. Across eleven depths and two
models it buys 1.11× to 1.23×. The per-step all-reduce has since been
timed, and it is not the bottleneck: it takes roughly a ninth of a step. On the same machine,
Qwen3-8B reaches 1.70× from its second card while paying the same collective. What holds these
two models back is not the interconnect; section 9 says what does.
At prefill the same card buys 1.56× to 1.87× in wall time, because attention parallelises with no communication at all.
Prefill is two terms and they do not move together. Fitting T(S) = a + bS + cS², against one 7900 XT the A100 leads by 3.3× on b and 6.7× on c for the dense 12B. The gap in attention scaling is twice the gap in compute. On the mixture-of-experts, where 4B active parameters shrink the compute term, the two gaps match at 5.75× and 5.71×. Datacentre silicon is not uniformly faster either: the L4 is slower than a consumer 7900 XT on b (0.90×) and 3.0× better on c. Datacentre versus consumer is not a single axis.
Evaluating the pair against one A100 is the third step, returning to where this article began. Without speculation, the A100 leads by 1.30× to 1.46× across all eleven depths of the ladder. That margin is flat throughout, and nothing like the U that four single-run points suggested.
Four machines enter the comparison: one RX 7900 XT, two of them at TP=2, one
A100-SXM4-80GB and one L4. Not every model runs on every
machine across the three architectures; two gaps stem from arithmetic rather than omissions. gemma-4-26B-A4B
stops at 12 000 tokens on one 7900 XT: holding 16.96 GiB of weights on a 19.98 GiB card leaves
only 0.93 GiB for KV cache even at 0.95 utilisation, whereas 32 K would need about 0.97. And
gemma-4-31B-it has no single-card row at all — 22 GB of weights do not fit into a 20 GB
card, so the pair is the only way this machine runs it. The 31B enters only at the third step
and appears nowhere before it.
The 31B step retains its original configuration: checkpoint
gemma-4-31B-it-qat-w4a16-ct, batch 1, and greedy sampling. Both platforms run on the
TRITON_ATTN attention backend. Heterogeneous head dimensions in this model force
Triton whenever FA4 is unavailable, which holds true for gfx1100 and for an A100 alike.
Prefill is fitted, not quoted. A single prefill tok/s number states nothing
because prompt scaling is non-linear: time to first token over S tokens follows T(S) = a + bS + cS².
The two primary machines differ by three times on one term and nearly seven on the other. Below,
fits come from `build_prefill.fits()`—the same ones prefill.jsonl --fits reports—over
chart-grade rungs only. The constant a is fitted but never quoted: it drifted from 9.8 ms
to 99.8 ms across two campaigns on the same arm, so this ladder does not measure it.
Begin without tensor parallelism: one 7900 XT, one A100, and one L4, evaluated across the same ladder on two models. The A100 leads throughout, by a margin that varies by model and contracts with depth. The L4 does not lead at all.
The L4 is a datacentre card that loses to a consumer card at this job. Across both models and every context depth, it reaches only 0.54× to 0.68× of one 7900 XT. What it exhibits instead is flatness: it retains 89 % of its 500-token decode rate at 32 K, whereas the A100 keeps 62 %. That stability is a property of a small card with slow memory, not an advantage. The takeaway from this comparison is that datacentre is not a tier, and the next figure is where the L4 earns its place.
The second 7900 XT forms the flattest line in Figure 1. Across eleven depths on the dense model it buys 1.13× to 1.23×, and across seven on the mixture-of-experts 1.11× to 1.12×. Scaling is not even monotone: on the 12B the ratio peaks at 1.23× at 6 000 tokens before dropping to 1.13× at 32 K.
A batch-1 decode step is bandwidth-bound and brief. Splitting model weights halves the bytes read per GPU, which is the whole gain. Every step then concludes with an all-reduce whose latency does not shrink with the split. This page used to conclude that what the second card contributes in bandwidth it largely spends on the wire. Timed, the wire is about a ninth of the step — 1.85 ms of 16.70 on the 12B — and a model on this same box gets 1.70× from its second card while paying the same collective. The collective latency is measurable, but it is not the limit; see section 9.
Prefill is the opposite case, and it is the same two cards. Attention over a prompt parallelises across heads without communication, and prompts are long enough that per-step overhead becomes a rounding error. Fitting both arms and recombining terms at the deepest rung each pair reached: 40.1 s to 21.5 s at 32 K on the dense model, 1.87×; and 6.21 s to 3.99 s at 12 K on the mixture-of-experts, 1.56×. Same hardware, same day, four to seven times the gain.
The linear term b represents GEMM throughput—how fast the machine multiplies. The
quadratic term c measures how severely attention scales with prompt length. A prefill tok/s
metric is simply a weighted average of the two at a single prompt length, shifting with every
prompt depth. Fitted separately, the two terms reveal contrasting behaviour across the same machines:
b is µs per
token of prompt and c is ns per token squared; both come from
build_prefill.fits() over chart-grade rungs, r² ≥ 0.9999 on every arm drawn. The
constant term is fitted and not shown, because this ladder does not measure it. Three readings emerge, and none survives being averaged into one number:
On the dense model the A100's attention advantage is twice its compute advantage — 6.67× against 3.29×. Whatever limits the 7900 XT at prefill, it is not primarily the multiplier.
On the mixture-of-experts the two collapse together, 5.71× against 5.75×. The model activates 4B of its 26B parameters per token, so the GEMM term stays small on both machines and the ratio between them ceases to reflect matrix multiplication.
The L4 is slower than a consumer card on b and three times better on c. 0.90× and 3.01×. That explains why it is 1.58× faster than a 7900 XT on a 32 K prompt yet loses on a short one: the two curves cross, making prefill advantage entirely a question of prompt length. The second card, for comparison, buys 1.44–1.48× on b and 1.91–2.22× on c — improving the quadratic term more than the linear term, mirroring the wall-clock findings above.
C-31B-tp2 of 2026-07-25 against the A100 ladder of 2026-08-30, from
benchmarks/decode.jsonl. Dotted and short dash are the 2026-08-29 pair, the only
session that ran both speculation arms; its Radeon side carries three patches the A100 side does
not, so it is drawn for what MTP costs rather than for the size of the gap. The 2026-08-26
probe switch adds the four single-run points this article was first written against — k=1, one
run a cell, and a deepest cell that compares the Radeons at 32K against the A100 at 30K because that
was the longest leg that side had. Its U is the shape of four points, and neither ladder behind it
has one. The gap is flat at about 1.4×. Running stock software without patches on either side, the A100 leads 1.36× at 500 tokens and 1.44× at 32K — narrowest at the shallowest depth, widest at the deepest, across a sixty-four-fold range of context. That answers the question this article's title used to ask, matching the comparison published in the repository README.
The lead is what speculation costs the A100. Speculation must be evaluated from the
2026-08-29 pair, because it is the only session that ran both arms on both machines — and its
Radeon side runs three patches the A100 side does not, so its level matters less than its
difference. Without speculation that session puts the A100 ahead 1.30× to 1.46×, close to
the stock ladder above it. Switch MTP on and the lead vanishes:
0.88× at 500 tokens, level from 2K through 8K, and only 1.08× at 32K. The Radeons do not get faster.
The A100 gives up more by speculating — and not because it drafts badly. Its acceptance is the
better of the two, 1.92–2.07 tokens a step against 1.46–1.53. It loses because a
speculative step costs it 2.40–2.57× a plain decode step, whereas the same step costs the Radeons
1.35–1.39×. At batch 1 the pair has spare compute inside each step; the single card lacks headroom, making the
fixed price of drafting and verifying four tokens a much larger tax on its work.
Both figures reside in benchmarks/campaign-2026-08-29/ and
benchmarks/cuda-a100/campaign-2026-08-29/.
The U was four points. Enable the probe and the curve appears — 1.48× at 1K, down to 1.14× at 16K, back to 1.87× at 32K — an anomaly this article was originally named for. Eleven matched rungs on the same two machines, averaged over two rounds a cell, produce a flat 1.36× to 1.44× instead. The 16K point that looked like a minimum was a single cell of one run, whose deepest neighbor compared two different depths. The shape did not survive replication; the raw points remain visible on the page.
And now at prefill too. The A100 side of this comparison lacked usable prefill until 2026-08-30: the session that measured its decode ran with prefix caching on, across a ladder whose every rung prefixes the next, charging repeated tokens for nothing and failing repeatability criteria. Re-taken with caching off across the five models running on both systems, prefill reveals dynamics invisible in decode:
c reaches 19 and b stops at 4 — because what is being compared is within a
term. The grouping is by the backend vLLM chose on the A100, which is recorded and checked against
the serve log beside it for all five; the Radeon side records one for three of the five and the
campaigns that did not keep a log carry null rather than a guess. Changed 2026-09-02: the 31B and
Qwen3.8 pairs' Radeon arms used to be the 2026-08-29 sitting, on which one of the two cards had
trained at PCIe 3.0 x8 rather than x16 (the host's boot journal shows it; the guest's sysfs cannot),
and a narrowed link moves b, which is half of what this figure divides. A reboot
restored the width and both arms were run again on the same container and the same serve arguments:
the 31B's b falls 868.7 → 722.6 and Qwen3.8's
846.2 → 758.5, so the 31B bar drawn here is 2.31× no longer
but 1.92×. Those are the sittings drawn. The 12B and 26B pairs are
2026-08-24, on x16. The compute term does not care which kernel is on the path, and the attention term is
almost nothing else. Across all five models the A100 leads by 1.92× to 4.00× on
b, with overlapping ranges between kernel implementations. On c they diverge completely:
2.99× to 3.10× on the three models the A100 serves through the same Triton
unified-attention family the Radeons use, and 11.94× and 19.01× on the two it serves through
FlashAttention.
Five models differ in more than their kernel, so that is a correlation. Here is the same effect
inside one machine, one day and one model. Qwen3.8-27B on these cards is routed to
ROCM_ATTN by default and can be pinned to TRITON_ATTN with a flag. Pinned,
its c is 17.27 ns/token² and the A100 is 11.94× ahead. Left alone, its c
is 4.46 and the A100 is 3.08× ahead. One flag, no patch, and the answer to "how far ahead is the
A100 on long prompts" moves by 3.87×. That flag was termed free by this repository
until 2026-08-30; it buys 1.15× of decode at 32 K and sells this.
None of which diminishes the A100: it remains ahead on every model and both terms. What it does mean is that a single number for “how much faster is the datacentre card” is not a property of the two cards: on the quadratic term it is mostly a property of which attention kernel each side ended up on, and on this hardware that is decided by a routing table and one command-line flag.
The 2D attention path—which speculation falls into under stock vLLM—launches one workgroup per KV head. This model has 16. On one card a rank sees all of them; split across two, each rank sees eight. The starved path is starved twice as hard by parallelism, visible directly in retention:
(total_num_q_blocks, num_kv_heads) at batch 1: the
workgroup count is a property of how the KV heads are split, not of the vendor. The same collapse,
the same kernel, and about twice the damage on two cards.
The MTP drafter is four layers deep, making the extra forward pass cheap. What is not cheap is that every draft step remains a step. Under TP=2, each step incurs the baseline all-reduce floor regardless of the model evaluated — measured at 2.30 ms on the 31B, invariant to drafter depth. Speculative decoding is therefore less advantageous on two cards than on one.
splitkv-31b-stock.json here, the backend matrix's forced-default column there — and both
are single runs. What is being compared is the gain, not the rates.
The nominal ratio assumes both sides reach theoretical peak bandwidth, which neither does. Recomputing realized bandwidth from the July campaign — decode rate times the weight bytes each GPU moves per token, against 800 GB/s — puts this comparison's model at 63 %:
benchmarks/results.jsonl at 500 tokens of context,
TP=2, against the per-GPU bytes per token stated in
benchmarks.md
§3. Two effects are visible and both are real: at equal quantisation the larger model
reaches higher utilisation, and the BF16 model reaches higher still. No equivalent
figure was measured on the A100, so the comparison in Figure 3 is against a nominal
ceiling on that side and a realized one on this side. Added 2026-09-02: every bar here
is derived — tok/s times the checkpoint's size, assuming a decode step reads every
weight byte once. Measured against hardware counters on an A100-SXM4-40GB, a decode step reads
81.6 % of the checkpoint on gemma-4-12B and 85.6 % on gemma-4-31B, so
the derivation overstates by 17–23 % there. It cannot be measured on gfx1100 — the VFIO guest's
counters read zero — so these bars stand as an upper bound.
Benchmarks run at batch 1. Batched throughput is a different regime and is not measured at all; where decode stops being bandwidth-bound,
the second card's 1.1× has no reason to hold, and neither does anything else on this page. Prefill is measured now, but only its two shape terms: the constant a is fitted and never quoted, because on one arm it moved from 9.8 ms to 99.8 ms between two campaigns a month apart and on another it came out negative. A short enough prompt is therefore outside what these fits describe.
The 31B has no single-card row and cannot have one on this box: 22 GB of weights, 20 GB of card. The first two steps rely on the other models; the 31B enters only at the third step where dual-card execution is required.
Figure 7 exists for Radeons with no counterpart on the other platform. The 1.27× marker in Figure 3 is a ratio of datasheet figures, an upper bound on what the bandwidth difference alone can explain, not a prediction either side was measured against.
Two vLLM versions run native stacks across environments, including rented Colab VMs with unquantified variance. Only one A100 and one L4 were measured once: nothing separates card architecture from instance effects.
Figure 3's solid line carries the comparison
using stock code on both sides. The 2026-08-29 pair is not patch-matched: its Radeon arms run
vllm#45916, the window block-skip and vllm#45450, while unspeculative A100
arms omit all three and speculative arms run only vllm#45450. With no unpatched run
covering both arms, MTP cost must be read off relative differences rather than absolute levels. Speculative
configuration matches: serve logs record SpeculativeConfig(method='mtp', …, num_spec_tokens=3) against
the identical assistant checkpoint, asserted by verify_doc_figures.py.
This page used to say that fixed all-reduce
latency ate most of the second card's bandwidth during batch-1 decode, and that no all-reduce had been timed to check it. One has been now, and it does not support that reading. Timed under HIP-graph replay—logged as cudagraph_mode FULL_AND_PIECEWISE and Capturing CUDA graphs (decode, FULL)—an
all-reduce at the batch-1 shape costs 16.6 to 21.5 µs. Across 2 × layers + 1 collectives
(tallied on 2026-09-03: two per layer plus embedding), communication takes 1.85 ms of the 12B's 16.70 ms
step, about a ninth.
And the same box gives Qwen3-8B 1.70× from its second card at decode
while giving the 12B 1.185×, on collectives that cost within 0.6 ms of each other
per step. Were the wire
eating bandwidth, the 8B could not reach 1.70×. What holds the 12B back is not the wire.
Subtracting perfect halving and the measured collectives
leaves 4.96 ms of its step unexplained; the same subtraction on the 8B
leaves 0.66 ms. (Answered 2026-09-02: the premise was flawed.) The model assumed memory
bottlenecking; yet on one card the 8B keeps the memory controller 90 % busy versus 56 % on
the 12B. Across five cells, scaling mirrors that counter alone (campaign-2026-09-02d).
The 12B's 4.96 ms is not missing time; it is the null model being wrong for a controller half idle.
That counter has since ordered the worth of faster memory, a second card and a slower card across four rented NVIDIA configurations, which is
its own article.
The half of the old sentence about attention still stands: attention parallelises with no communication at all, and the prefill numbers above are what show it. The three
claims pricing collectives from fitted intercepts remain withdrawn: they were unmeasurable on this ladder,
replaced by direct timings 49 to 63 times smaller.
The
measurement is in the repository, with the runtime library identified from /proc/self/maps rather than a version constant.
README.md. The 2026-08-29 pair remains to evaluate speculative decoding
arms, now explicitly identifying its patches.
README.md, conflating the 12B metric with
the 31B, which reaches 62.8 %. Corrected and dated in place, all three bandwidth
utilisation figures are now recomputed by verify_doc_figures.py directly from campaign logs.
The data. Benchmark traces reside in
benchmarks/speculative-decoding/ for Radeons,
benchmarks/cuda-a100/45450-validation/ for
A100 runs and port logs, and
benchmarks/results.jsonl for Figure 7 records.
Every figure is recomputed from those files by
verify_doc_figures.py.
Why both stacks required a patch to be worth comparing is the previous article.