RDNA3 · MoE · torch.compile · measured on 2× RX 7900 XT

Run under vLLM's --enforce-eager, the fastest model here recorded 107.8 tok/s as 15,
and invented two findings on the way

Run under --enforce-eager at 15 tok/s; checked against torch.compile on two cards, with power symmetric and decode context-dependent

An earlier benchmark run gave torch.compile twenty minutes before timing out. Adding --enforce-eager recorded about 15 tok/s, leading this repository to conclude that MoE was impractical on ROCm. Engine initialisation actually requires twenty-six minutes. Compiled, the same model is the fastest of five architectures measured on this machine.

2× RX 7900 XT · gfx1100 · TP=2 gemma-4-26B-A4B · 128 experts Zen 1 host · 8 cores measured 2026-07-25

Abstract

When compiled, a 128-expert MoE decodes at 107.82 tok/s at 500 tokens of context and 72.80 at 32000, leading every dense model measured here, including checkpoints with higher parameter counts. Under --enforce-eager the same model was recorded at about 15, and that unoptimized figure formed the basis for the published verdict.

Avoiding eager mode requires an engine startup of 1569 seconds, of which 54 is loading weights. Compilation runs single-threaded on an eight-core host because vllm/env_override.py assigns TORCHINDUCTOR_COMPILE_THREADS = "1" on import. Because it is a hard assignment, exporting the variable in the environment has no effect.

The 7.2× throughput loss is not the main damage. Eager execution also produced two qualitative artifacts that were mistaken for architectural traits: a 131 W / 65 W power split across cards, interpreted as expert imbalance, and a decode rate flat across context, interpreted as hybrid-like scaling. When compiled, the 22 measured rows are symmetric to within 2.26 % with identical VRAM on both cards, and throughput falls by a third across the context ladder.

Both erroneous conclusions were published in this repository before being corrected. The upstream fix is two lines — vllm#53891 and vllm#53892 — which preserve the single-thread default while allowing an explicit environment override.

1What is being measured

The campaign from 2026-07-25 covers seven configurations and eight engine starts, totaling 146 timed generations across five models and two parallelism settings. The server reports decode throughput over a generation of several hundred tokens after a warm-up; each cell represents the mean of two rounds. Board power is sampled during each generation and recorded per card as the maximum seen in that window, alongside per-card VRAM.

init_engine_s is engine initialisation, not compile time. It encompasses loading weights, profiling, allocating the KV cache, torch.compile and graph capture. The weight-loading share is tracked separately, which is what lets Figure 2 show that on the two slow starts it is not where the time goes. Nothing here isolates Inductor's own share of the remainder.

The eager raw output is not in this repository Every compiled figure below is recomputed from benchmarks/results.jsonl. The eager measurements lack committed raw records: they come from an earlier probe whose raw output was never committed, surviving only as quoted text in docs/benchmarks.md. This article extracts them from that document rather than restating them, so the two cannot drift apart, but extraction is not the same as recomputation and the figures say so where it applies. The number that decided an architecture verdict is the one with the weakest provenance in this article.

2The verdict that got published

The exploratory probe logged the model as "torch.compile did not finish in 20 minutes → impractical → forced --enforce-eager → ~15 tok/s", adding "severely asymmetric power draw (131 W / 65 W), the MoE fails to load both cards evenly". The model was filed under architectures held back by ROCm, and the repository's architecture table read "mediocre, ~15 tok/s" for months.

Nothing about that initial reasoning is unusual. A compile that has not finished in twenty minutes looks like a compile that is not going to finish, and eager is the documented way past it. The step that does not survive is the subsequent assumption: treating what eager then reports as a measurement of the architecture.

Figure 1 · where each architecture ranks at 500 tokens of context recomputable from repo eager bar not committed
Figure 1. Decode rate at the shortest rung of the ladder, mean of two rounds, from benchmarks/results.jsonl. The striped bar is the same model under --enforce-eager; it is drawn differently because it is the one value in this figure that cannot be recomputed from the repository. Sorted, the MoE appears at the top of the same list and near the bottom of it. The hybrid-SSM entry at the bottom is a different story with two causes of its own and is here only to place the ladder.

The 12B dense model repeated the mistake independently. An earlier context sweep used --enforce-eager to dodge the same compile wall and produced "a flat 15.8 tok/s, independent of context, with power bouncing between the cards". Compiled it reaches 59.91, and the artificial flatness is gone.

3Where twenty-six minutes goes

Figure 2 · every engine start in the campaign recomputable from repo
Figure 2. All eight starts, in the order they ran. Two of them exceed twenty minutes and four are under two; the remaining pair, the 31B and the 27B, sit between. On the two slow ones the weight-loading share is 3.4 % and 0.4 % of the total. Whatever is expensive, it is not the model coming off disk. The one repeated configuration is the 12B at TP=1, started twice eight minutes apart: 59.67 s then 33.36 s. That is the only cold-against-warm pair the campaign contains, and it is not one of the two slow ones.

The reason it is one core is a single line, run on every import vllm:

vllm/env_override.py:105 · still this on main, checked 2026-08-29
os.environ["TORCHINDUCTOR_COMPILE_THREADS"] = "1"

Because the code uses direct assignment rather than a setdefault, any value exported in the environment is overwritten before Inductor reads it. Importing vLLM forces torch._inductor.config.compile_threads == 1 whatever the caller asked for. Inductor's own default would allocate one worker per core.

The default itself is not the problem Single-threaded compilation is deliberate upstream, addressing edge cases recorded in vllm#10480 and vllm#10619: compile subprocesses interact badly with debuggers and specialized launch paths. Changing that default is a different argument from the one made here. What is reported is narrower: the mechanism provides the operator no way to opt out. That omission was not intended; a plain assignment is simply what a setdefault looks like before an override is needed.

4The two things eager invented

A 7.2× throughput loss is an obvious measurement flaw that a reader can discount. The deeper error is that eager produced two qualitative results, each reading as a fact about the model rather than about the flag.

Figure 3 · both fabricated phenomena, against the compiled run recomputable from repo eager row not committed
Figure 3. Board power per card at each rung, compiled, from the same 22 rows as Figure 1's MoE bar; the two ticks are the two cards on a common 0–300 W axis. They sit on top of each other at every depth, worst case 2.26 % apart, and VRAM is identical on both cards in every row. The red row is the eager report the imbalance conclusion was drawn from, at 50.4 % apart. The second view is the same run's decode rate, which falls to 67.5 % of its short-context value — eager's flat line was the flag, not the architecture.

The power reading had an intuitive explanation ready, but it was wrong. Under pure tensor parallelism vLLM maintains all experts on every GPU and shards each expert's weights along the intermediate dimension, so compute remains symmetric whatever the router does. Routing imbalance is a property of expert parallelism, which was not running here. The asymmetry disappearing under compilation confirms it: nothing about expert execution changed.

5What was sent upstream

Two lines, and they do not touch the default:

vllm#53892 · 2 insertions, 1 deletion, 1 file
-os.environ["TORCHINDUCTOR_COMPILE_THREADS"] = "1"
+os.environ.setdefault("TORCHINDUCTOR_COMPILE_THREADS", "1")
reported
vllm#53891, 2026-08-26, labelled rocm: an exported TORCHINDUCTOR_COMPILE_THREADS is silently overwritten.
proposed
vllm#53892, same day. The default of one is preserved; explicit exports are respected.
verified
On the ROCm 7.14 / vLLM 0.23 image, importing vllm.env_override with the variable exported to 8 leaves compile_threads == 8; unset, it resolves to 1 as before. Unpatched, the custom export is clobbered.

6What is not established

That the compile output is usefully cached

Guidance suggests a 26-minute compile is amortized by caching, but the campaign cannot verify this. The sole repeated start is the 12B at TP=1, dropping from 59.67 s cold to 33.36 s warm — a 1.79× saving on a start that was never the problem. Neither configuration exceeding twenty minutes was restarted here, leaving warm-start costs unmeasured. Compile caching is real upstream, but unverified on this host.

How much of the 1569 s is Inductor

init_engine_s aggregates loading, profiling, KV allocation, compilation, and graph capture. Weight loading is isolated and negligible. The remaining phases are not decomposed; "the compile takes 26 minutes" is shorthand for "engine init takes 26 minutes and it is not the weights".

The eager side of every comparison

The eager rates and the 131 W / 65 W reading lack committed raw logs. Quoted from documentation notes, a re-run under --enforce-eager is needed to put them on equal footing with the rest of the campaign. The performance direction is certain; the exact ratio holds single-digit precision at best.

One host, and it is a slow one

The benchmark host is a Zen 1 Threadripper with eight cores allocated to the container. The compile-thread argument scales with core count, giving high-core systems proportionally larger gains. The patched build's compile time was not measured directly, pending upstream adoption.

7What has changed since

2026-07-25 · the verdict was reversed in place With the startup timeout raised the compile finished, and the architecture table went from "🟡 mediocre, ~15 tok/s" to "🟢 fastest measured here". The revision is dated in docs/architecture-notes.md rather than written over the original, because the earlier entry is the point.
2026-08-26 · reported and proposed upstream vllm#53891 and vllm#53892. As of 2026-08-29 both are open; the PR is mergeable and waiting on review, and line 105 on main is still the plain assignment.
2026-08-29 · three numbers in this repository did not match its own data Writing this article checked the compile figures against results.jsonl for the first time. The 12B's compile was quoted as 23 minutes in one document and as 24 minutes at TP=1 in another; the committed value is 1537.92 s at TP=2, and TP=1 took 59.67 s. The "1538 s cold, 33 s warm" pair put a TP=2 cold start next to a TP=1 warm one. All three are corrected and dated in place, and verify_doc_figures.py now pins every engine-start time in the campaign so they cannot drift back.
2026-08-29 · a count in this article was wrong The English version of the paragraph above said seven engine starts. The campaign has eight, across seven configurations: the 12B at TP=1 was started twice, which is the repeat the warm-start figure is drawn from. The Chinese version had it right. Corrected in place; verify_doc_figures.py was already checking the count against model_meta and the prose was the only place it was retyped.
2026-08-29 · figure 2's caption miscounted its own bars It said two starts exceed twenty minutes "while the other six are under two". Four are under two; the other two are the 31B and the 27B, which the figure prints at 227 s and 504 s directly above the sentence that denied them. Corrected in place, and verify_doc_figures.py now pins the three counts so the caption cannot drift from the bars again.

The data. Benchmark data resides in benchmarks/results.jsonl, logging decode rate, power, and VRAM per generation, plus engine start metadata. The figures here are recomputed from it by verify_doc_figures.py; the uncommitted eager row is extracted from docs/benchmarks.md §1.

Upstream: vllm#53891 and vllm#53892. Upstream context on the single-thread default is documented in vllm#10480 and vllm#10619.