Read as max_seqlen_q; measured against no-speculation at 71 % loss on a Radeon and 61 % on an A100
Turning on vLLM's own documented MTP assistant makes gemma-4-31B decode
36.9 % faster at 1K of context and 70.8 % slower at 32K. The cause is not the
drafter, not the hardware, and not the acceptance rate. Speculation sets max_seqlen_q = 2.
One clause in an or chain reads that length as prefill rather than decode, dropping the kernel from
128 workgroups to 8.
vLLM documents this assistant as supported and provides an example command line without mentioning context length. Measured across four depths, the throughput change is monotonic and changes sign: +36.9 %, −14.0 %, −32.4 %, −70.8 %. By 8K, speculative decoding is already a net loss.
The gate is max_seqlen_q > 1 in the Triton unified-attention backend's
use_3d condition. The 3D kernel splits a long KV sequence into segments and reduces them
in parallel. Speculation's second query row disqualifies that path. The launch grid collapses from
1 × 8 × 16 to 1 × 8. The code comment attributes this clause to prefill requests in the
batch, but the binding constraint is buffer geometry: the segment buffer is allocated per sequence and
indexed per token, dimensions that coincide only at one query row.
Three explanations that look better were measured and dropped. Not KV-head starvation: this model has 16 heads and decodes cleanly without speculation. Not query row overhead: calling the kernel directly, moving from one row to two costs 0.08 % at 32K in one construction and 0.16 % in a second. Not tiling constants: porting upstream's Blackwell-tuned block made it 8.7 % slower.
The claim targets a kernel rather than specific hardware. It predicts that any CUDA system routed to this kernel will reproduce the same collapse as context grows, whereas backends lacking the guard will not. A rented A100 tested both: −28.2 % at 30K and −61.1 % at 50K on the forced default, and the collapse gone on FlashInfer. Both predictions held.
The benchmark evaluates gemma-4-31B-it-qat-w4a16-ct at TP=2, batch 1, and greedy decoding,
paired with the official gemma-4-31B-it-assistant at num_speculative_tokens: 1. Differencing
a 64-token generation against an 8-token one at the same depth isolates decode from prefill.
benchmarks/speculative-decoding/. Monotonic across four
depths, and it crosses zero between 1K and 8K — the crossover was not located.
Hover any point for both rates. The MTP arm was measured a second time twenty-five days later,
as the stock baseline of a different comparison. The three long depths reproduce to 0.45
% and 1K differs by 6.0 %, so the end of the ladder that carries the finding is the end
that repeats best.
vLLM's internal counter at 32K records Mean acceptance length: 1.12, representing 9 accepted
tokens out of 78 drafted. Back-solving from the 1K speedup implies roughly 0.46 acceptance there, showing that
acceptance degrades with depth as well. Rejected drafts carry a real cost: each pays for the assistant's forward
pass and target verification. Yet lower acceptance cannot account for −70.8 % when the assistant spans only four layers.
Section 5 shows what can.
use_3d = not ( seq_threshold_3D is None or ... or max_seqlen_q > 1 # speculation makes this 2 or num_seqs > seq_threshold_3D or is_batch_invariant )
What that clause decides is how much of the GPU the kernel is allowed to use:
The comment on the clause attributes the guard to prefill requests in the batch. The binding
constraint is buffer geometry, which indicates what a real fix requires. The segment buffer
is allocated as [seq_threshold_3D, num_heads_q, segments, headdim]—per sequence—and
indexed as segm_output[token, head, segm_idx, :]—per token. With one query row
those coincide. At two the buffer would overflow, so the guard excludes multi-row queries, and
max_seqlen_q > 1 catches speculation alongside prefill because it cannot tell them
apart.
The intuitive explanation is that two query rows simply demand more work. To test this, direct benchmarks
called unified_attention using the model's exact tensor shapes without model weights or speculation,
sweeping the number of query rows:
Two more explanations were tested and eliminated. KV-head starvation: the collapse in
the hybrid-SSM article comes from having 4 KV heads, but this
model has 16 and decodes cleanly without speculation. Tiling: upstream carries a
tuned_large_head block for head_size == 256 and max_seqlen_q > 1, gated to
Blackwell, with every other condition matching this hardware. Porting its BLOCK_M = 32 made
32K 8.7 % slower.
kernel_unified_attention at 32K, from torch-profiler traces stored on
the test machine. Because only the summary and trace filenames are committed, this is the one block on
the page that cannot be recomputed:
Most calls barely move and a minority explode. That divergence reflects this model's alternating attention: the
sliding-window layers see a short KV and are unaffected, whereas global layers scan the full 32K and forfeit
segmentation entirely. Quoting the 9.2× mean would describe neither population.
hipGraphLaunch goes 63 to 324, confirming graph replay is intact.
A claim about a kernel generates testable predictions for hardware it was not measured on. If the collapse
stems from dropping from 3D to 2D execution, any CUDA system routed onto the same kernel should replicate it,
deepening with context, while backends without the guard should not. Because gemma-4 has heterogeneous head
dimensions, vLLM forces TRITON_ATTN whenever FA4 is unavailable—and an A100 has no
FA4. That makes the prediction testable for the price of an hour of rented time.
benchmarks/cuda-a100/gemma4-mtp-backend-matrix.json, vLLM 0.28.0, one run per
cell — a probe, not a campaign. The grey mark is this machine's −70.8 % at 32K, for scale.
On the forced default, MTP is a net loss that deepens with depth; on FlashInfer the same MTP is
positive at 30K, and the residual −8.7 % at 50K is drafter economics rather than the pathological
drop. The two 30K FlashInfer figures come from different Colab VMs with an identical stack; every
other ratio here is computed within one session, and the 50K quartet is entirely one session.
Widening the buffers by hand and raising the cutoff recovers 32.42 tok/s at 32K against 8.85, so speculation finally outperforms baseline decode. That was not a permanent fix: it cannot express "speculation's two query rows but not prefill's four thousand", and greedy output on a 30K prompt diverged from stock at token 35, with three possible causes unseparated.
vllm#45450, open since June and
stalled with merge conflicts, provides the structural fix that paragraph asked for. The gate becomes
max_seqlen_q > decode_query_len, so prefill stays excluded while segment buffers
are sized per token.
The divergence question is settled too. A fixed-input comparison at the kernel level across 18 cases shows each path bit-deterministic across the 18 fixed-input cases, with the two differing by at most one bf16 ULP at one query row—today's shipped behaviour—exactly as at two and four. The token-35 divergence reflects pre-existing 2D/3D reduction-order reassociation already toggled by batch size in production, not a defect of the admission.
The results above apply when the Triton unified-attention kernel serves the model. vLLM does not always route execution there, and when it routes elsewhere, this patch sits idle. The same A/B, five times control on Figure 4 draws the five runs that separate the two cases, each eleven rungs and two rounds a cell at k=3:
gemma-4-31B-it on an A100, TRITON_ATTN, probe fires once —
+45.6 % on average, 1.95× at 32K.gemma-4-26B-A4B on an A100, TRITON_ATTN, probe once —
+48.4 %, 1.99× at 32K.Qwen3.8-27B on the two Radeons with the kernel pinned by
--attention-backend, probe twice, one per tensor-parallel worker —
+83.9 %, 2.87× at 32K.Qwen3.8-27B on the same Radeons as the engine routes it, ROCM_ATTN,
probe silent — −1.9 %: the two arms are one measurement taken twice.Qwen3.8-27B on an A100 as the engine routes it, FLASH_ATTN, probe
silent — −0.08 %. At 32K, 20.51 against 20.52.The probe identifies which case applies before running the ladder.
inject_45450.py prints PROBE_3D_SPEC_ACTIVE once per worker process when
the 3D path serves a step wider than one token, requiring three concurrent conditions: the patch
installed, this kernel active, and speculation enabled. In all five runs the probe and outcome
agree: where it printed, the patched arm pulled away; where it stayed silent, no rung moved by more
than 8.8 %, within normal run-to-run variation. A reader who applies this patch and observes a silent log
has not mis-applied it. The engine simply routes that model elsewhere, making the backend the proper fix rather
than the patch.
The same 3D-to-2D drop was documented first in
vllm#48076 on 2026-07-09, on a
single B200 running the same architecture in a different quantisation. The trigger in that report is
a different clause in the same or chain: num_seqs > seq_threshold_3D,
a batch-size threshold whose proposed fix derives the threshold from SM count. That fix
does not reach this trigger. Because max_seqlen_q > 1 operates independently of the
threshold, speculation falls to 2D regardless of how the threshold is computed. What is new here is
the second trigger, its underlying cause, and measurements on RDNA3.
The finding was posted to that thread on 2026-08-01, and on 2026-08-26 to vllm#52049, a CUDA-side report of the same symptom on the same model. That reporter confirmed that explicitly selecting FlashInfer sidesteps it, matching our analysis: the collapse traced here is specific to this Triton kernel, so choosing an explicit backend bypasses the forcing. ROCm lacks FlashInfer, so on this hardware the practical remedy remains gating speculation by depth.
What is not specific to this kernel is speculation losing at depth. Measured on
2026-08-29, Qwen3.8-27B loses too on ROCm's ROCM_ATTN, whose backend file
imports chunked_prefill_paged_decode and none of the files this patch edits, and on
CUDA's FLASH_ATTN. Neither backend uses this kernel, and on neither does this patch change
anything. They share the same sign through a different mechanism that this article does not identify. It would be a
mistake to read those two as evidence for the argument above; they demonstrate that the
argument above does not cover them.
Between 1K and 8K, unmeasured. On the A100, 30K and 50K were measured, bracketing where FlashInfer MTP ceases to pay rather than locating the exact crossover.
The ROCm ladder uses one process per cell. The MTP arm has a replicate twenty-five days later agreeing to 0.45 % at the three long depths, but the no-speculation arm lacks replicates and per-cell acceptance rate was not captured. The A100 matrix is likewise single-run per cell across two VMs with unquantified cross-session variance.
All measurements use temperature=0, where draft and target agree most often. Sampling
should lower acceptance and degrade speculation, but was not measured on either vendor.
Tested on gfx1100 at TP=2 (a VFIO guest lacking peer-to-peer, one checkpoint) and on one A100, one card, one checkpoint. The source-level kernel claim is supported by both machines agreeing, not by either measurement alone.
docs/speculative-decoding-on-rdna.md §1 said every figure in it is a single run with no
process replicates. The MTP arm has one: the stock baseline of the vllm#45450 comparison is the same
configuration measured again, and it agrees to 0.45 % at 8K, 16K and 32K. The document now says so, and
verify_doc_figures.py pins the spread.
The data. The ROCm ladders, direct kernel sweeps, and correctness cases are
in benchmarks/speculative-decoding/;
the A100 matrix, logs, and vLLM 0.28 FlashInfer override recipe are
in benchmarks/cuda-a100/.
The full write-up is
speculative-decoding-on-rdna.md.
Profiler traces of 2 MB per call stay on the test machine, with derived distributions and filenames
in trace-unified-attention.json.
The assistant is google/gemma-4-31B-it-assistant, 0.90 GiB and not gated. Its
head_dim and global_head_dim match the target, enabling shared KV cache allocation.