SUPPORTED_QUANT_TYPES adds a flat 60 ms per step,Asymmetric and symmetric decode rate on dual Radeon cards under vLLM; checked across context lengths, against native kernels, and against upstream fixes
A 27B model decoded at 12 tok/s on this machine. Every document here treated that as one fault with one cause. It is two costs stacked on each other: one flat, one growing with context. Separating them takes a single A/B. Once separated, each turns out to have its own upstream fix, written by someone else.
The same model family, packaged asymmetrically and symmetrically, differs by 3.241× at 1K of context, 1.786× at 8K and 1.266× at 32K. Read as a ratio, that looks like an effect fading. Read as decode time per step, it is the opposite: the difference stays between 58.03 and 62.79 ms, a spread of 8.2 %, while the ratio falls by 60.9 %. A flat cost sits under a growing one.
The flat cost comes from the W4A16 linear path. An asymmetric checkpoint arrives as
uint4. One line of SUPPORTED_QUANT_TYPES admits only
uint4b8, so the layer silently falls to Triton. Nothing warns, and the model name does not
indicate which format is present.
The growing cost is paged attention, which
has its own article. Both halves have an upstream fix, and
neither fix is ours: vllm#40977
merged, adding a second native kernel that accepts uint4;
vllm#45916 widens the split-KV
attention gate and is still unmerged.
What this article contributes is the separation, and one finding uncovered on the way: sweeping the asymmetric configuration space on gfx1100, three of twelve configurations are served by two kernels, one by Triton, and eight by nothing at all. Every asymmetric checkpoint with act-order is in the last group.
Decode rate here is a differenced measurement, not a server-reported one: generate 8 tokens, generate 64, and divide the extra 56 by the extra time. That removes prefill and the first-token latency from a number meant to reflect steady-state decoding. Every cell runs in a fresh container off a stock image, following a discarded warm-up generation.
The warm-up is necessary rather than ceremonial. Every cell JIT-compiles exactly four Triton kernels. On both arms all four compilations land inside the warm-up, so neither timed call carries a compile. That is checked directly from container logs rather than assumed, and it is checked on both arms so a compile cannot inflate one side only.
Which kernel actually served the layers is recorded from inside the workers. Under
--tensor-parallel-size 2 the linear layers are built in spawned processes, where a
monkeypatch applied in the parent sees nothing at all. So choose_mp_linear_kernel is patched
on disk before the engine starts and each rank writes its own verdict. Every cell in this article carries
two distinct pids; a cell whose instrumentation never reached the workers would otherwise write an empty
record and exit zero.
On 2026-08-24 this box measured two 27B models on the same day: a dense baseline at 44.8 tok/s and this hybrid-SSM model at 12.3, a factor of 3.64× with parameter count held constant. That was read as an architecture result, and it is the reading earlier repository documents carried.
The comparison was confounded. Every quantized checkpoint in that campaign is symmetric except the two Qwen3.x AWQ ones, and those are exactly the two the documents single out as slow. The architecture comparison and the packaging difference coincide precisely where the conclusion was drawn. The A/B below is what separates them, and it is worth up to 3.24× on its own.
Two checkpoints evaluate the same model architecture — Qwen3_5ForConditionalGeneration,
compressed-tensors pack-quantized on both sides, same two cards, same image, one fresh container per
cell. One is symmetric: false, the other symmetric: true.
Decode time per token is additive. If the cause is a per-step GEMM, it must appear as a context-independent number of milliseconds. In that case the ratio falls only because the attention term grows underneath both arms. That is a prediction the ratio on its own cannot make, and it is the reason this figure defaults to milliseconds rather than to the larger-looking multiple.
benchmarks/w4a16-symmetry/w4a16-ab.jsonl. In the millisecond view the orange block
is the difference between the arms and is drawn to the same scale in all three rows, which is
the argument: it does not grow. The ratio view is the same six numbers read the way a headline
reads them. One container start per cell, so no run-to-run range is quoted
here. The one repeat this experiment has moves the symmetric arm by at most 1.31 % across
container starts, so the 8.2 % spread in the penalty is larger than the noise floor but the same
order as it.
vLLM has had a native gfx1100 W4A16 kernel since vllm#41394, registered for the same mixed-precision path compressed-tensors uses. Its own docstring places it ahead of the Triton kernel on the ROCm-RDNA3 path. One line decides who reaches it:
SUPPORTED_QUANT_TYPES = [scalar_types.uint4b8]
uint4b8 is symmetric int4. compressed_tensors_wNa16.py picks the weight type
from the checkpoint's own symmetric flag and sets zero_points = not symmetric.
An asymmetric checkpoint therefore arrives as uint4 and fails the type check. Asked directly, the
registry says exactly that:
no RDNA3W4A16LinearKernel Quant type (uint4) not supported by
RDNA3 W4A16 kernel; supported: [uint4b8]
SELECTED TritonW4A16LinearKernel
Nothing warns. The model loads, runs, answers correctly, and is slow.
w4a16-selection-2x2.json). It follows the symmetric flag and ignores the group size,
which is what retires the obvious confound — the two arms also differ at 32 against 128. Bottom:
every quantized checkpoint on this box, with the kernel the registry picks for its real
quantization parameters. The fastest model measured here shares its group size with the slowest, and
gemma-4-26B-A4B-AWQ is named AWQ and is symmetric, so the name is not a guide. The field
that decides is quantization_config.config_groups.*.weights.symmetric.
Two other differences between the arms had to be retired rather than argued away. Group
size, 32 against 128: no checkpoint exists that separates it from symmetry, so this A/B cannot.
However, can_implement has no group-size-128 condition in its source, the four corners above
ignore it, and the fastest model on this box is group 32. Which layers are quantized:
the two ignore lists differ, so the quantized linears were counted from each
model.safetensors.index.json rather than assumed. 399 against 400, differing by exactly one
layer that only the symmetric checkpoint quantizes — which gives the symmetric arm one more
W4A16 GEMM, so it cannot be what makes that arm faster.
The flat block is the whole of the difference between the arms, but it is nowhere near the whole
of the decode step. At 32K both arms spend most of a step inside
kernel_paged_attention_2d. Its per-call cost grows by a factor of 28 between 1K and
32K while every other kernel in the profile stays flat. That is the second problem: it is not about
quantization at all, and it has its own article and its own
fix.
These two terms explain why neither headline number was ever a good summary. At 1K the packaging penalty is essentially the entire gap; at 32K it is a rounding error next to attention. A single ratio quoted without its context length hides which of the two it is mostly measuring.
vllm#40977 merged on 2026-07-14 and
added RDNAHybridW4A16LinearKernel, which accepts uint4, group size 32, and all
of gfx11. On current main an asymmetric checkpoint does get a native kernel — a
different one from the kernel this investigation was about, reached by widening the registry rather than
the type gate. The same checkpoint that decoded at 11.41 tok/s against Triton decodes at 37.32 there.
vllm#45916 provides the attention half. This repository's contribution to it is the gfx1100 end-to-end verification rather than the code.
benchmarks/ledger.jsonl. The step from 0.23.1 to
0.27 crosses ROCm, vLLM and max_num_seqs at once, so it is a statement about stacks
rather than about one variable; the controlled measurement of the packaging penalty is Figure 1, not
this arrow. A hollow ring after a value marks a cell the ledger declines to grade for chart use.
Hover any point for its run count and range. A cross-campaign control makes the 0.27 arm worth trusting on its own terms. The 1K stock cell was
measured twice by the split-KV campaign and once more, weeks apart and by a different script, as the
hybrid arm of the kernel pair in this directory. The two campaigns land
0.21 % apart. Neither was corrected towards the other.
The type gate can be opened. Doing it exposes a second problem behind the first, which is the part
worth reporting: the checkpoint ships its zero points as (N/8, groups) and the kernel wants
(groups, N/8), so the layer selects, loads, and then refuses at the kernel entry with
b_scales must have same group count as qzeros. The HIP kernel was never reached with wrong
data; it declined the shapes at the door.
Three Python lines fix both halves: admit uint4, run the transpose that
process_weights_after_loading already runs on the other two tensors, and pass the
zero-point convention the kernel has always supported instead of hard-coding the GPTQv1 one.
3.11× on 0.23.1, with output token for token identical to the Triton control.
terasterasterasteras…. A patch that stopped after the layout fix would have looked like a
success on every axis except correctness.
On main it is still the wrong patch. Where Hybrid competes it wins by 5.9 % (37.32
against 35.22), and RDNA3W4A16LinearKernel sits ahead of Hybrid in the ROCm
priority list, so admitting uint4 there hands the layer to the slower of two working
kernels. The patch cannot tell that case apart from the case where it is the only option.
coverage-gap.json). This is can_implement only — no kernel
runs — so it says which configurations load, not which are fast. Every asymmetric configuration with
act-order has no kernel at all on gfx1100 today: Hybrid rejects g_idx, Triton rejects it
too, and choose_mp_linear_kernel raises. desc_act=True is a standard GPTQ
quality option, not an exotic setting.
The eight are checked numerically rather than left at the registry's word: a real desc_act
permutation against an fp32 reference that indexes scales and zero points through g_idx,
fp16 and bf16, decode and prefill shapes. Four cases fail before the patch, four pass after, and the full
suite goes to 54 passing with no regression.
That is where this sits. Not a pull request, because taking the nine without the three needs a decision that belongs to whoever owns the priority list, and the same argument was already made from the gfx1151 side on vllm#46186. Widening Hybrid instead is probably what upstream wants, and that is not our code to change.
The additive decomposition is consistent with a per-step cost that does not depend on context, and the kernel record says which GEMM ran. It does not by itself exclude some other flat term riding along with it. What is measured is that the inter-arm gap remains flat to 8.2 % over a 32× context span, and that distinct linear kernels serve the two arms.
These are independent quantizations by two publishers at different group sizes; this measures speed only. "Prefer a symmetric checkpoint on RDNA3" is a throughput statement. Whether the two models match in quality is not established here.
One machine, one model family, gfx1100 only, TP=2, bf16 activations only. The
partition_scales and channel-wise paths are untouched, and the permute assumes a group-quantized layout. The coverage sweep is a registry check on synthetic parameters; which of those nine configurations anyone actually ships is not measured.
Six cells, one container start each. The existing repeat covers the symmetric arm only, moving it by at most 1.31 %. The 0.27 arms in Figure 3 carry two runs each with their own ranges.
RDNAHybridW4A16LinearKernel. On main an asymmetric checkpoint reaches a
native kernel and the "falls back to Triton" framing applies to 0.23.x and to builds without that PR.
Everything about RDNA3W4A16LinearKernel itself — the type gate, the transposed zero-point
layout, use_v2_format — is unchanged on main.
README.md, docs/benchmarks.md
and docs/architecture-notes.md were corrected and dated, and
docs/open-questions.md §9 is marked answered. What survives untouched: every slope
conclusion, and the ordering among the four models that are all on their best kernel path. c.zero_points. That passed the end-to-end
run but broke ten existing cases, because a GPTQ checkpoint stores zero points explicitly and still
uses the v1 "+1" offset. The right key is not c.weight_type.has_bias(). Keying on
zero_points would have silently mis-dequantized every asymmetric GPTQ checkpoint on
RDNA3 while passing the run that was supposed to validate it. docs/benchmarks.md says so — "unmerged, so on a stock install this stands". Section 9
had it right and the abstract did not. Corrected in place.
The data behind every figure. The A/B run, forced-kernel data, three-line fix
and its control, coverage sweep, and upstream tests are in
benchmarks/w4a16-symmetry/,
together with per-cell worker kernel records. Every figure on this page is recomputed from
those files by
verify_doc_figures.py,
which exits non-zero if any figure disagrees.
Upstream: vllm#40977 is the fix for the flat half and vllm#45916 for the growing one; vllm#50264 is where the second was reported. The trade-off between widening the RDNA3 kernel and displacing Hybrid was argued from the gfx1151 side by @JartX on vllm#46186.