Hybrid-SSM decode rate on dual Radeon cards under vLLM; checked against dense controls, llama.cpp, and an upstream split-KV kernel
Linear attention is supposed to make long context cheap. On this machine
Qwen3.6-27B falls from 12.1 to 4.2 tok/s between 500 and 32 000 tokens, while a
dense 8B on the same afternoon keeps 77 % of its short-context rate. The 48 linear-attention
layers are not the problem. The 16 that are left are.
A hybrid Mamba/gated-delta-net checkpoint loses 65 % of its decode rate between 500 and 32 000 tokens of context under vLLM on RDNA3, a curve fourteen to forty times steeper than any dense model measured on the same machine in the same campaign.
A kernel-level profile puts essentially all of the growth in one place:
kernel_paged_attention_2d. That Triton fallback serves the model's 16 full-attention
layers, and its per-call cost grows 28.3× while every other kernel in the step remains flat or slightly
cheaper. The linear-attention layers are not where the time goes.
The ROCm custom paged-attention kernel declines this model on three separate conditions, forcing execution onto the Triton fallback. The fallback is slow on this model in particular because 4 KV heads at batch 1 leave the kernel almost nothing to parallelise over. Two independent signals agree: per-card power falls with depth, and normalising by KV bytes read makes the gap wider rather than narrower.
The same checkpoint under llama.cpp on the same machine and the same day holds a slope 28× shallower, which rules out the driver and the hardware. An upstream fix exists and is measured here twice, most recently as a controlled A/B on vLLM 0.27: 9.46× at 32 K. One cell of that A/B is bimodal and is reported as such.
Everything in §2 comes from one campaign: one software stack, one afternoon, every model pinned to it.
That run produced 292 decode and prefill measurements across five architectures and eleven context lengths. Each cell
is served by vllm serve, TP=2, CUDA graphs enabled, --gpu-memory-utilization 0.85,
512-token outputs, and is the mean of two rounds.
Rates are reported with the spread of their own rounds rather than a standard deviation. At two to four rounds per cell a standard deviation provides more precision than the sample supports. The range as a fraction of the mean expresses variance cleanly without implying a distribution. Every point in Figure 1 carries its round count and that range, visible on hover.
Five architectures, eleven depths, one campaign. Four of the five decline gently with context in the way a KV cache makes inevitable. The fifth does something else.
Qwen3.6-27B, the hybrid-SSM checkpoint this article is about,
and Qwen3-8B, the dense control. The other three are one click away. Retention from 500 to
32 000 tokens: 8B 77.2 %, 12B 69.9 %, 26B MoE 67.5 %, 31B 68.4 %,
hybrid-SSM 35.1 %. The last two lines are the successor checkpoint on the two kernels
the engine can serve its full-attention layers with, measured 2026-08-29 without speculation on
either; they are off until clicked because they are a different stack from the campaign above.
Source: benchmarks/ledger.jsonl.
The hybrid model is the slowest in absolute terms at every depth. By parameter count it should not be. It is also the only model whose curve bends downward instead of flattening. Retention makes the contrast plain: every other configuration keeps between 67 % and 78 % of its short-context rate at 32 000 tokens. This one keeps 35.1 %.
Expressed as a slope — decode microseconds added per token of context — the campaign gives 4.84 µs for the hybrid model against 0.118 µs for the dense 8B. That is a factor of 41, and it is the number the rest of this article is about.
A torch profile across two depths kept CUDA graphs on and attributed kernel calls strictly to the engine step whose window they fell into, so that chunked-prefill calls are not mixed into the decode figures.
The 48 gated-delta-net layers cost the same at 32 K as at 1 K — 0.95× — which is what linear attention promises and what this implementation delivers. The quantised GEMM is likewise flat. One kernel moves, and it moves by 28.3×.
The arithmetic closes. At 16 calls per step, paged attention goes from 5.71 ms/step to 161.5 ms/step, up 155.8 ms. The decode step itself, measured from the CUDA-graph replay, goes from 85.85 ms to 236.9 ms, up 151 ms. The paged-attention growth accounts for the entire decode slowdown; every other kernel is flat or slightly faster at the longer context.
The 16 full-attention layers land on Triton because ROCm's custom paged-attention kernel declines them.
The RDNA branch of use_rocm_custom_paged_attention in vllm/platforms/rocm.py
requires, among other conditions, head_size == 128 and block_size == 16.
# measured on the running engine, not read off the config head_dim = 256 # gate wants 128 cache_config.block_size = 784 # = 16 x 49, gate wants 16 is_pow2(784) = False # fails use_custom on its own path gqa_ratio = 6 # this one passes # forcing either failing condition on its own is still not enough use_rocm_custom_paged_attention(head_size=128) -> False use_rocm_custom_paged_attention(block_size=16) -> False
No single condition is the blocker. Relaxing either failing condition on its own
leaves the gate closed, and chunked_prefill_paged_decode.py fails use_custom on
is_pow2 independently of both. The custom kernel is unreachable here three times over, which
is why making the fallback fast is the only path that helps this class of model.
head_size check moves the failure rather than removing it.
CALL_CUSTOM_LAUNCHER_BLK_HEAD in csrc/rocm/attention.cu dispatches only
case 64 and case 128 and TORCH_CHECKs on anything else. The
condition is the kernel's actual limit, not conservative gating.
Some of the slope is the kernel, not the architecture. The successor
checkpoint, Qwen3.8-27B on vLLM 0.27, was measured on 2026-08-29 against both of the
kernels the engine will serve those 16 layers with. There was no speculation on either arm: same checkpoint,
same session, both repeating to better than 0.12 %. Left to itself the ROCm selector takes
ROCM_ATTN, because TURBOQUANT is incompatible with the decoder and that is
the first fallback it reaches for. Pinned to TRITON_ATTN instead — one flag, no patch —
the same ladder is 15.0 % faster at 32K, and retention across the whole span goes from
73.4 % to 84.3 %. Eleven points of the collapse this article is about belong to which
kernel serves those layers. The rest does not: even pinned, the model still gives up 15.7 % of its
short-context rate by 32K, which leads into the next section.
Dense models on the same machine take the same Triton path and do not degrade like this. The difference is how much parallelism the kernel has to work with at decode, where the batch is 1.
| model | KV heads | query heads | GQA | µs per context token |
|---|---|---|---|---|
| Qwen3-8B, BF16 dense | 8 | 32 | 4:1 | 0.118 |
| gemma-4-31B, w4a16 dense | 16 | 32 | 2:1 | 0.339 |
| Qwen3.6-27B, hybrid | 4 | 24 | 6:1 | 4.840 |
With 4 KV heads and a batch of 1 there is very little for the kernel to spread across the card's 84 CUs. Two independent signals agree that this is occupancy rather than work:
Total power draws 265 W + 265 W at 518 tokens, then drops to 232 W + 227 W at 24 K. No other model in the sweep loses power with depth. Every other model climbs toward 265 W and stays there; this is the only one that comes back down. Baseline levels differ — at 518 tokens the 8B sits at 245 W, the 12B at 206 W, and the 26B MoE at 193 W — so the contrast that matters is the direction, not the level. Idle silicon, not saturated silicon.
Only 16 of the 64 layers carry a KV cache, and their GQA ratio is aggressive. As a result, this model reads the least KV data of the three: 64 KiB per token against 144 KiB for Qwen3-8B and 160 KiB for gemma-4-31B. Per KiB read it reaches 92× the cost of Qwen3-8B and 36× that of gemma-4-31B. It cannot be that this model is simply heavier.
Same machine, same day, same checkpoint family, only the inference engine changed. If the collapse were in the driver, the GPU, or the topology, it would appear here too.
benchmarks/llamacpp-depth-sweep-{rocm,vulkan}.json,
benchmarks/results.jsonl.
llama.cpp's ROCm slope lands inside the band the dense models produce under vLLM; its Vulkan slope is below the band, flatter than any of them. Two independent sources therefore agree on what a normal slope looks like on this hardware, but this one configuration sits an order of magnitude outside it.
Prefill, meanwhile, runs the other way for this configuration: throughput improves with length, 802 → 881 tok/s from 500 to 32 K, a rise of 9.9 %, while every dense arm measured beside it loses 36–44 % over the same range. It is decode that inverts. (Corrected 2026-08-31: this said 8–44 %. The 8 was the 8B, whose 500-token rung is not chart-grade — its two rounds are 22 % apart — read at the better of the two, which is what makes its fall look small. On the rungs that pass the cut it falls 37 % from 1 K, and no dense arm here falls less than 36 %.)
That rise belongs to this configuration, not to the architecture. It read as the
architecture's here until 2026-08-31, on two hand-typed numbers from one model on one machine.
Qwen3.8-27B is the same hybrid-SSM design, and on this box its prefill does not rise: it
falls 11.8 % under vLLM 0.23 and 32.6 % pinned to Triton, and on the one arm re-measured on a
full-width link it is flat, +0.3 %; on an A100 it falls 8.1 %. (Until 2026-09-02
this sentence quoted 7.5 % for that arm. That was the same run with one card trained at PCIe 3.0 x8;
campaign-2026-09-02c is the re-measurement, and it moves the arm from falling to flat,
not to rising.) (Corrected 2026-09-03: this sentence used to say that this checkpoint's
prefill never rises. On an H100 it rises 19.4 % across the same rungs — further than
the Qwen3.6 arm this section is about. The claim was written about a model and was only
ever true of the machines the model had been run on.) Meanwhile gemma-4-26B-A4B, a
MoE with no linear-attention layer at all, rises further than Qwen3.6 does, 23.8 % in
July and 21.9 % in August. Of the twenty-four stock hybrid-SSM ladders now in
benchmarks/prefill.jsonl — one checkpoint family across ten machine configurations —
ten rise by more than 1 %, one is flat, and thirteen fall. The same
Qwen3.8-27B checkpoint runs from −68.3 % on two Radeons — the 2026-09-07
ladder to 128 000, pinned to TRITON_ATTN — to +103.0 % on two H100s. One architecture and one
checkpoint produce both signs across a range of 171 points. That is the core finding: what a rising prefill curve indicates is that a short
prompt's fixed cost outweighs the quadratic term on that arm, not that the model carries a
recurrent state. The decode collapse this article is about is unaffected either way:
it is measured, not inferred from the same reasoning.
The upstream fix is vllm#45916, which adds a split-KV decode kernel to the Triton fallback: instead of one workgroup per (sequence, KV head) walking the whole context, the KV range is split across more workgroups and reduced afterwards. That is exactly the parallelism this model was short of.
It was verified here first on vLLM 0.23.1 in July — 2.52× at 32 K in single differencing runs at slightly different depths. Then properly, as a controlled A/B on the 0.27 image: both arms on one stack and one harness, one fresh container per cell, only the kernel file swapped, run twice with the arm order reversed.
Qwen3.8-27B-AWQ-INT4, TP=2, vLLM 0.27.1.dev5, offline 64-vs-8 differencing.
Different model and different stack from Figure 1 — see the caveat below. Routing was recorded
from inside the TP workers, not inferred: (head_size 256, sliding_window 0, use_splitkv True)
on both ranks at every depth in both passes, and both file md5s are asserted at runtime so an arm that
silently failed to swap would abort rather than measure the other arm twice.
Source: benchmarks/hybrid-splitkv-027/.
Pooled across both passes: 1.38× at 1 K, 3.52× at 8 K, 9.46× at 32 K. Decode goes from 261.9 to 27.7 ms per token at 32 K, the slope from 7.408 to 0.262 ms per 1 000 tokens of context, and retention from 1 K to 32 K from 10.2 % to 70.0 %. The collapse is gone.
Four limits.
At 8 192 tokens the patched arm ran 47.02, 40.14, 47.58, and 42.41 tok/s over four passes. Sorted, the middle gap spans over half the range: these are two modes about 15 % apart, not noise. Passes three and four widened the span from 15.8 % to 16.8 %. The deterministic split heuristic cannot cause this; no mechanism is established. The quoted 3.52× mean fits neither cluster.
Figure 1 tests Qwen3.6-27B on 0.23.1; Figure 4 tests Qwen3.8-27B on 0.27. The newer stack picks another W4A16 linear kernel, giving roughly 3× on its own without attention. Absolute levels in the two figures are not comparable; ratios and shape are.
The causal A/B ran on one box, one vendor, at batch 1 throughout. Decode ladders to 128 000 tokens on four rented NVIDIA cards show Qwen3.8-27B falling 21.8 % on an H100 (matching dense 31B). Attention sets depth cost across vendors, but backends varied, precluding clean same-kernel control. The occupancy argument predicts the collapse should soften as batch size rises, and that is untested anywhere here.
Paged attention drops from about 68 % to 11 % of decode time. Quantised GEMM dominates the rest, taking roughly 77 % at short context. The slope is resolved; baseline throughput is a separate question.
This article is a dated record, not standing advice. Three items have moved since the measurements above.
on_gfx12x() to on_gfx1x() locally so that gfx11
could reach the new kernel. That change was merged into vllm#45916's own branch on 2026-08-10, so the PR
now admits RDNA3 as it stands and nothing local is required.
verify_doc_figures.py now pins the ratio and each backend's position against the band.
Data and reproduction. Figures marked recomputable from repo
are verified against source data by
verify_doc_figures.py,
which fails non-zero if prose and data disagree. Background context is in
docs/hybrid-decode-on-rdna.md.
Raw rows: ledger.jsonl ·
hybrid-splitkv-027 ·
repository.
Reported upstream as vllm#50264. The fix is vllm#45916 by @feiyehua, with the gfx11 widening by @thegoldenflow; this work contributed the gfx1100 end-to-end verification and the 0.27 A/B.