methodology · decode throughput · 2× RX 7900 XT

An offline probe matches the campaign harness to 0.44 %,
after the first of four runs reads 31 % low

Decode throughput on a pair of Radeon cards; checked between an offline probe and the campaign harness, across cold starts, and across thirty days of drift

Two harnesses in this repository produce decode figures: an offline probe and the campaign harness. Charts that pool their results need the difference to be measured rather than assumed. On a warm machine, that difference is 0.44 %. Finding that out required four identical runs, because the first read 31 % low.

2× RX 7900 XT · gfx1100 · TP=2 calibrated on Qwen3-8B · bf16 170 ledger rows measured 2026-08-26

Abstract

The campaign harness queries an OpenAI-compatible server with streaming, sampling at temperature 0.8 over a 512-token generation. The probe harness runs LLM.generate offline with greedy decoding, differencing a 64-token run against an 8-token run. At matched context depths, the two agree to 0.44 %, 0.01 % and 0.07 %. Matching the generation span changes nothing of consequence. They record the same measurement.

Reaching that conclusion required discovering that a single run is not a measurement on this box. Four identical runs back to back read −30.7 %, −5.1 %, and then converge. The penalty falls heaviest where the machine is fastest, which initially disguised a warm-up effect as a harness discrepancy.

Two rules follow and are recorded in the ledger rather than left to whoever plots the charts. Every point records runs and range_pct. patches is a list rather than a boolean, because a result requiring an unmerged pull request is not the same claim as one obtained from a release.

The range is reported rather than a standard deviation, and one cell demonstrates why. Adding a third and fourth run to a bimodal 8K point widened its range from 15.79 % to 16.79 %, even as its standard deviation fell from 4.87 to 3.61. One statistic warned that the measurement was less stable; the other suggested it had improved.

1Do the two harnesses measure the same thing

The calibration model is chosen specifically to isolate harness differences from every other variable in the stack:

Engine knobs follow the campaign configuration. Context depths use the campaign's reported prompt_tokens rather than nominal targets, ensuring both harnesses measure at identical sequence lengths rather than merely near them.

Figure 1 · the campaign's number against the probe's recomputable from repo

Figure 1. The campaign column is the mean of its two committed rounds for this model on 2026-08-24; the probe column is the mean of the third and fourth calibration runs, which is where it converges. The last column repeats the comparison with the probe generating 512 tokens instead of 56, which is the campaign's own span. It moves the numbers by at most a further 0.5 point, so the span is not where any difference lives, and neither is the server, the streaming, or the sampling temperature.

2The three runs that cost, and what they bought

The initial comparison in Figure 1 showed the probe 31 % below the campaign at the shortest depth and 9 % below at the longest. Because the two harnesses differ in token span, sampling strategy, and transport, that gap invited plausible explanations. It was none of them.

Figure 2 · four identical runs, back to back recomputable from repo

Figure 2. Same container image, same script, nothing changed between them. The first run was the first container started after stopping the machine's other GPU services, and it was slow for its whole four minutes rather than only at its first depth, so it is not the probe's own warm-up generation failing to warm. The mechanism is not established here. What is established is the size and the shape: it converges by the third run, and it hurts the fastest points most — which is exactly why it read as a harness difference before the repeats existed.

Why this does not retroactively damage the campaigns Each campaign cell launches a fresh container whose engine initialisation — loading the model, compiling kernels, capturing graphs — runs for minutes before any timing begins. Campaign cells for this model already execute two rounds that agree to 0.2 %. In a separate experiment, reversing the arm order yielded agreement of 0.3 % at 1K and 0.2 % at 32K, consistency that a first-cell penalty of this magnitude would have shattered. The resulting rule applies specifically to probe runs launched on a cold machine: discard the first run rather than averaging it in.

3The control that spans thirty days

Two campaigns ran thirty days apart, under different kernels and different patches. Six configurations were measured in both. Those six controls are the sole evidence determining whether a number from July and a number from August can share a single chart.

Figure 3 · the same configuration, measured twice, thirty days apart recomputable from repo

Figure 3. Mean percentage difference across every context depth the two campaigns share, August against July. Four of six sit inside a quarter of a per cent. The two that do not are not dismissed: one of them is the subject of a separate investigation which ruled out temperature, the w4a16 path and PCIe atomics, and which has not identified the cause. These two are the reason to trust the other four.

4What every point carries, and where the line is

The ledger projects raw measurement files into a single uniform row format, recomputed and verified rather than edited by hand. Two metadata fields accompany every point; delegating them to chart rendering is how a benchmark repository promises more certainty than it measured.

Figure 4 · every point's run-to-run range, and where the cut sits recomputable from repo

Figure 4. One mark per ledger row that has more than one run, on a log axis because the distribution spans three decades. The cut is not a round number chosen in advance: it sits in the gap the data leaves. Points above it are drawn with a hollow marker in every chart and named in the caption.

One afternoon provides the clearest illustration of why this metadata must accompany the data. The 2026-08-29 campaign evaluated eight ladders on a single machine across eleven shared rungs, running two rounds per cell in one continuous session. The three non-speculative arms placed zero of their 33 rungs above the cut, recording a worst range of 0.59 %. By contrast, the five speculative arms placed 28 of 55 rungs above the threshold. That is not a statement about speculation as such: gemma-4-31B's speculative arm is graded at every rung, worst range 2.68 %. Instead, all 28 outliers belong to Qwen3.8-27B — on both attention backends, patched and unpatched alike. Reporting a simple mean would have presented those 28 volatile cells as placidly as the other 60, concealing the instability in every downstream plot.

5Why a range and not a standard deviation

Almost every cell in this benchmark contains two runs. With two samples, the sample standard deviation is merely the range divided by the square root of two — the same metric masquerading as an unearned distribution. At four samples, the statistic begins to carry independent meaning. That is where the choice becomes concrete, because one measured cell has four runs.

Figure 5 · one bimodal cell, two statistics, opposite conclusions recomputable from repo

Figure 5. The split-KV arm at 8K on this model produces two clusters rather than a spread. Adding the third and fourth runs made the range wider, which is the correct signal: the cell is less trustworthy than two runs suggested. The standard deviation moved the other way, because a fourth point near the lower cluster makes the sample look more evenly distributed. At this sample size the range is the statistic that cannot flatter a cell it does not understand.

6Why output comparison is not the correctness test here

The obvious way to verify that a kernel change is harmless is to compare generated tokens before and after the modification. On this hardware, however, that test measures platform nondeterminism rather than the change.

Figure 6 · greedy decoding is not reproducible across processes on this box recomputable from repo

Figure 6. Greedy, temperature zero, a warm-up generation of the same prompt at the same depth before the measured one, and no code differing between the runs of a cell. It varies across processes and inside one process, symmetrically between the two kernel states, so the change under test is not the cause. The kernel itself is not the cause either: fixed-input cases run in separate processes agree bit for bit. Whatever varies is above the kernel, in the engine, and it is not identified.

Numerical claims in this repository are evaluated against an fp32 reference on fixed inputs. End-to-end changes are argued from throughput rates and internal execution routing logged by the workers. Comparing token IDs is reserved exclusively for platforms known to be deterministic — a property confirmed by measurement on the A100, but absent on this machine.

7What is not established

Why the first run is slow

The slowdown is measured, but the mechanism is not. It outlasts a warm-up generation, is not confined to the first depth, and clears by the third run. Whether the cause is the host, the driver, the container or something else was not isolated.

Whether the harnesses agree on other models

Calibration uses one model to isolate variables. Agreement is clean but narrow: models on other kernel paths could diverge between harnesses for reasons unaddressed here.

The two controls that missed

Two of six configurations shifted about 0.8 % between campaigns. One remains under investigation after ruling out several candidates without finding the cause. Cross-campaign pooling is therefore bounded by a band, not asserted as equivalence.

What varies in the greedy output

Varying outputs are not claimed to be wrong, only non-reproducible. TP=1 is untested; variance originates above the kernel but is not further localised.

8What has changed since

2026-08-26 · the first calibration attempt was the interesting one It compared one probe run against the campaign and produced a 31 % harness difference with a plausible explanation available. Three more runs turned that into a warm-up effect. The rule this article exists to state — that a single run is not a measurement here — comes from nearly publishing the opposite.

2026-08-29 · the cut's justification was restated build_ledger.py justified its 6 % threshold by describing the spread distribution as it stood when the rule was written. That distribution moved: the 0.27 rows landed and the bimodal cell gained two runs, so the tail was then 2.6, 3.4, 3.5, 4.1, 5.1 and then 16.8, and the cut excluded one point rather than two. The comment was rewritten to describe it and the verifier pins its tail, so it cannot go stale unnoticed.

2026-08-29 · and again, with the cut moved The ladder campaign landed 88 more points the same day, 55 of them speculative. The tail is where they gather: of the 29 points now above the cut, 28 speculate. The twenty-ninth does not — it is the bimodal 8 K cell from the day before, at 16.79 %, worse than most of them. The pin above caught the move, which is what it is for. The tail now runs 5.30, 5.33, 5.97, 6.02, 6.10 and then jumps to 9.50, so 6 % no longer sat in a gap: it stranded two rungs between itself and the real break. The cut is 8 %, the middle of that break. It regrades exactly those two, both measured the same day, so nothing published before it changes.

2026-08-29 · two intervals and one scope Section 2 said "the campaigns already run two rounds per cell which agree to 0.2 %". That is true of this calibration's model — the Qwen3-8B cells' worst spread is 0.216 % — and not of the campaigns at large, where benchmarks/README.md counts 30 of 142 cells above 1 %. The sentence is now scoped to the cells it is about. Separately, the two campaigns are thirty days apart, 2026-07-25 to 2026-08-24, not five weeks; the heading, the paragraph and figure 3's title all said five. And the Chinese rendered "stated with a band" as 带宽陈述, which is bandwidth; it now reads 区间.

2026-09-02 · the attention backend is excluded as the source of the non-determinism Section 6 says the greedy outputs this machine produces vary across processes. Holding the W4A16 kernel at RDNA3W4A16LinearKernel and forcing TRITON_ATTN on the two unstable models left both unstable, against zero varying cells for the two the earlier reading called stable; the backend is not the axis, and the quantisation kernel is the remaining named candidate, not established (gfx1100-greedy-attn-ab/).

Run it yourself. All data and verification scripts live in the repository: probe runs in benchmarks/harness-calibration/; the ledger pipeline in ledger.jsonl and build_ledger.py, which fails if the committed projection no longer matches its sources; and raw nondeterminism logs in gfx1100-greedy-nondeterminism.json. Every figure is recomputed from these sources by verify_doc_figures.py.

The nondeterminism was posted upstream in vllm#50603, which reports identical Triton fallback behaviour but finds that a warm-up call fixes it. Because every measurement here already included a warm-up generation and still varied, either that remedy does not generalise or this platform encounters a distinct defect.