RDNA3 · paged attention · vLLM · measured on 2× RX 7900 XT

vLLM's custom ROCm paged-attention kernel wins all sixty cells,
and the two bands overlap

Checked as use_rocm_custom_paged_attention; custom kernel against Triton fallback, with speedups of 2 to 7 times

On gfx11, vLLM's ROCm custom paged-attention kernel is enabled only when the query-to-KV head ratio is at least 3. In the excluded range, the custom kernel is 1.70× to 7.28× faster than the Triton fallback it is passed over for. Inside what bf16 gives for a sum this long, it is numerically equivalent. Sixty cells, two vLLM versions, no exception.

2× RX 7900 XT · gfx1100 vLLM 0.23.1 and 0.27.1 plus an A100 running the same Triton text proposed as vllm#54210 measured 2026-08-28

Abstract

The bound came in with vllm#17004, whose own description says the kernel "supports gqa_ratio up to 16, and shows performance gains over the existing kernel when gqa_ratio is 3 or higher". The guard is therefore a performance heuristic, not a capability limit. On CDNA, the same function runs this custom kernel starting from ratio 1. Both statements remain true on main.

When tested, the heuristic is inverted. Across sixty measured cells spanning five shapes, six context lengths, and two vLLM generations, the custom kernel is faster in every one. The excluded band overlaps the admitted band on both versions. There is no region the bound separates.

The report that prompted this, vllm#50603, also blames the fallback for losing accuracy at long context. That hypothesis is not supported. Across 90 cells on two architectures, relative error stays between 2.12e-3 and 4.72e-3, and no two depths' pooled medians differ by more than 1.09×. A flat band only means something if the instrument can see a fault, so the same harness was given one: NaN in the unwritten tail of the final KV tile poisons both paths, at exactly the lengths that straddle it and no others.

End to end on a real model, widening the gate is worth 1.02× at 1K and 1.17× at 32K, reproduced in both run orders on the newer image. That change is proposed in vllm#54210, and this page is what it rests on.

1What is being measured

Both paths are driven through vLLM's own chunked_prefill_paged_decode dispatch with only use_rocm_custom_paged_attention forced. Argument marshalling is therefore identical between the arms. Pure decode is measured: one query token per sequence over a fully cached context, which matches the shape described in the report. The fp32 reference is computed from pre-paging KV tensors rather than the cache, so a paging bug cannot cancel itself out. Scoring uses relative L-infinity against the reference's own scale, making different context lengths comparable.

The shape set covers ratios 1, 2, 3 and 4 at head size 128 and block size 16, plus 32/16; that final shape belongs to gemma-3-27b-it, satisfying every condition of the gfx11 gate except the ratio. For the CUDA control, the kernel is lifted verbatim into a standalone module compiling the same Triton text, and the launch mirrors the dispatch's argument list.

2Sixty cells, and no exception

Figure 1 · Triton time ÷ custom-kernel time, per shape and depth recomputable from repo

Figure 1. Above 1.0 means the gated-out kernel is the faster one. The rows the gate admits are the control: they show the advantage it was written to capture. The rows it excludes show the same advantage, at the same magnitude, being declined. Hover any cell for the two times behind it. The newer image is a little lower in twenty-eight of the thirty paired cells. Both arms moved between the images and this comparison separates neither; the two 0.27 rounds are 2.3 % apart at the median and 5.8 % at worst.

The bands are the argument, not the individual maxima. On 0.23.1, the excluded ratios span 1.84× to 7.28×, while the admitted ones span 2.35× to 7.40×. On 0.27, excluded ratios span 1.70× to 6.05× against 2.20× to 6.05× for admitted shapes. The two overlap on both versions, which is what "the bound does not separate anything" means in one sentence.

3The accuracy claim, and the control under it

The originating report describes two symptoms. The first symptom—taking the fallback path at all—is confirmed and forms the subject of the findings above. The second claim is that the fallback degrades with context. That hypothesis is directly testable here, and it does not hold.

Figure 2 · median relative error against context length recomputable from repo

Figure 2. Medians over all shapes at each length; the shaded zone is the full band across all 90 individual cells. The custom kernel sits consistently a little above Triton, which is a different accumulation order rather than a defect — both are inside what bf16 gives for a sum this long. The third series is the identical Triton source compiled on an A100, and it lands in the same band. That series is what makes this a bounded negative rather than an absence of evidence.

A clean sweep is worth nothing if the instrument is blind Two conditions went unexercised in the sweep above: every tested length is a multiple of the block size, so the final KV tile never straddles the sequence end, and the cache is zero-filled. Filling the cache with NaN and selecting lengths that straddle produces the fault on demand — and only there.

Only NaN propagates: finite garbage at 1e4 is handled correctly by the score mask alone, which is why this is subtle. Both paths poison identically on this version.

4What widening it is worth on a model

Kernel microbenchmarks alone do not justify a pull request. An end-to-end model test was run using gemma-3-27b-it-w4a16 at TP=2 with CUDA graphs on. Decode latency was isolated by 64-against-8 differencing, launching one fresh container per cell so source edits could not leak between arms:

Figure 3 · one line, one model, three depths, three passes recomputable from repo

Figure 3. The two 0.27 passes ran the arms in opposite orders — one measured stock first at every depth, the other widened first — so drift over the forty minutes cannot land on the same arm twice. Same arm, same depth, across the two passes: at most 2.8 % apart. The gain grows with context because of what moves: this model is 62 layers with a sliding-window pattern of 6, so only the ten full-attention layers can reach the custom kernel at all. At 1K the sliding layers dominate; at 32K a full layer scans the whole context while a sliding one scans 1024.

Routing is recorded directly inside the TP workers rather than inferred from timing. Measurements covered every cell on 0.23.1, and on 0.27 at 1K in both arms from a dedicated run after the in-place recorder returned nothing. The full-attention layers arrive with head size 128, block size 16, native KV layout, and a power-of-two block, so the ratio is the only gate condition they fail; the sliding layers fail on sliding_window == 0 in both arms and never move.

Two things this measurement is not Output equality between arms is not the correctness test. Because greedy decoding on this machine is not reproducible across processes for this model, comparing token ids would measure local nondeterminism instead. The numerical case is Figure 2's, against an fp32 reference. And the initial kernel counter was blind, because TP=2 runs attention in spawned workers where a parent-process counter sees nothing.

5What widening it would also admit

Opening a gate lets in whatever lurks behind it. Upstream PR vllm#53856 fixes a real fault in the same custom kernel: it masks logits past the sequence end, but its vectorised V-cache load still consumes the whole final block, allowing an unwritten NaN in the tail to poison output. Its test plan notes that gfx11 received compile validation only, because the author had no such device.

Figure 4 · the fault, and the fix, on gfx1100 recomputable from repo

Figure 4. 64 rows, 32 per arm: two dtypes × two ratios × four poison modes × two lengths, against the shipped image and against a rebuild carrying the PR at its current head. The pattern is the discriminating part rather than the pass count. K alone never poisons — the logit mask already covers it, so a fix that sanitised K would treat a symptom that is not there. Only the straddling length: 4096 fills the final tile exactly and is clean, 4090 leaves six unwritten slots and is not. Which kernel ran is counted by wrapping the op, not inferred from timing.

The forced column bears directly on the proposal. Forced past the gate, the stock arm poisons eight cells rather than four: ratio 2 fails identically at the straddling length across both dtypes, under both V-only and K+V corruption. All eight are clean on the patched arm. So widening the gate admits ratio 2 to this kernel, and this is the fault it would admit with it — which is why the two threads belong together, and why the gfx11 runtime evidence went to that PR's author on request.

6What is not established

The report's second symptom

These tests eliminate only one hypothesis: that the Triton 2D kernel loses accuracy as context grows. The reporter observed degradation on a VLM not available here, running on different hardware; this probe evaluates the kernel rather than model output. The reported symptom still needs an explanation.

Everything outside pure decode at these shapes

Sliding-window shapes, fp8 KV, prefill shapes, and kernel-level TP>1 were not tested. While the model benchmark uses TP=2, that is a model-level measurement rather than a kernel one.

Where the NaN comes from

Because allocate_kv_cache zero-fills memory, the defect is not uninitialised allocation garbage; vllm#53856 attributes it to sleep mode and allocator behaviour. This harness injects NaN deliberately to observe how the kernel responds.

One machine, and the model runner question

All measurements ran on a single gfx1100 machine. A related question — whether Model Runner V2 reaches the same guard — was measured here: it does, on 36 of 37 rows identical to V1. Whether speculation makes decode carry multiple query rows under MRv2 was not measured, because no available checkpoint configures for it on that image. That half rests on a code read.

7What has changed since

2026-08-28 · proposed upstream vllm#54210, one line in the gfx11 branch of use_rocm_custom_paged_attention: one insertion, one deletion. Open as of 2026-09-09: a maintainer asked on 2026-09-07 for kernel correctness and a full gsm8k run, both supplied the same day; no formal review yet. The PR body carries the tables on this page; the repository carries the rows they are computed from.

2026-08-28 · the gfx11 half of a different PR was supplied on request The author of vllm#53856 had validated the fix on MI350X and asked for gfx11, which they do not have. Figure 4 is that, at their current head, with both arms run because a clean patched arm proves nothing unless the stock arm reproduces the fault.

2026-08-27 · the speedup floor was understated It was first written as 2.06×, which is the minimum over ratio 1 alone. The real floor is the 32/16 shape at 4K, 1.84× on 0.23.1 and 1.70× on 0.27. Corrected and dated in place, and recomputed by verify_doc_figures.py rather than transcribed.

2026-08-29 · one figure needed its derivation stated, not corrected The source directory says a 16× context increase moves the median by 1.06×, which looked wrong against a ladder that spans 32×. It is not: that figure pools the two shortest depths against the two longest, whose centres are about 16× apart. Read end to end instead the move is 1.03×. Both derivations are now written down and both are checked, because the sentence was defensible but unreadable at the same time.

2026-08-29 · figure 1's caption attributed a move the data does not attribute It said the newer image sits "uniformly a little lower because the Triton fallback got faster, not because the custom kernel got slower". Neither half holds: two of the thirty paired cells are higher on the newer image, and across the thirty the Triton time falls in 21 while the custom kernel's time rises in 25. Both arms moved. The caption now says so, and verify_doc_figures.py pins both counts.

2026-08-29 · the routing claim covered one of the two stacks Section 3 said routing is recorded from inside the TP workers "in every cell". That holds for the 0.23.1 sweep. On 0.27 the in-place recorder produced nothing — the workers ran unmodified bytecode while the file on disk was modified, which benchmarks/vllm-50603/README.md records — and routing there comes from route_027.py, a run of its own at 1K in both arms. The sentence now says so.

Every cell. The probes, output JSONL, and container logs reside in benchmarks/vllm-50603/, along with the CUDA control's lifted kernel and the hash check keeping both sides on the same Triton source. Every figure here is recomputed from those files by verify_doc_figures.py.

Upstream: vllm#50603 is the report, vllm#17004 introduced the bound, vllm#54210 is the one-line change, and vllm#53856 is the fault Figure 4 is about.