KFD asks for write access on a read-only copy,
and an Ubuntu kernel backport charged a second for it

1The question, and what it was worth asking from

The opening measurement is the weakest in this article, but it prompted all the work that followed. It goes first, with its provenance attached. On 2026-07-24, with the disk measured at 1.5 GB/s by dd iflag=direct on the same file, a 15.6 GiB BF16 checkpoint loaded at 77 MB/s and a 9.56 GiB w4a16 one at 29. That was nineteen and fifty-two times slower than the medium it came off.

The first hypothesis was wrong, but stating it explains the method that followed. This repository published "the source is a file-backed mmap page, and ROCm does not take the pinned-staging path for those". A dependency-free reproducer disproved it. A plain mmap of a warm file copies at full speed. Eight variables — anonymous against file-backed, faulted against not, fresh range per copy, overlayfs against ext4, fresh device allocation per copy, a 1–256 MiB size sweep, page-unaligned starts, dtype — all failed to trigger anything. After that, the bar for any second explanation was that a stranger could run it.

2The variable that does trigger it

Two mappings of the same file, in one process, while safetensors holds a checkpoint open:

75d757200000-75d845520000 rw-p ... model-00001-of-00005.safetensors   <- safetensors
75d468a00000-75d556d20000 r--p ... model-00001-of-00005.safetensors   <- ours

The trigger is a read-only checkpoint mapped MAP_PRIVATE and writable. Isolating that one variable — same file, same bytes, plain mmap and torch.frombuffer, without safetensors or vLLM — reproduces the whole thing in sixty lines. Adding a second dimension, whether the CPU has touched the mapping before the copy, produces three regimes rather than two. Real loads sit in the third.

16 019.6 ms is not a bandwidth figure and it is not noise. It is sixteen 1000 ms HMM_RANGE_DEFAULT_TIMEOUT windows plus 19.6 ms of real work. The sub-millisecond reproducibility that looked like a fixed per-page cost is a jiffies timeout. Once you read the number that way, the rest follows: the flat one-second gap between two filesystems is one extra retry, and per-tensor timings of 1005, 1042 and 1522 ms are one, one and one-and-a-half windows.

3Why a read asks for write

AMD confirmed the mechanism on ROCm#6523 and named the site. The chain is three hops in the upstream tree:

The permission is taken from the VMA rather than from what the GPU is about to do with the range. A host-to-device copy only reads its source. Because the VMA carries VM_WRITE, KFD asks hmm_range_fault() for write access, which breaks copy-on-write on every resident page. That is the permanent defect. It appears on every kernel measured here, and the line predates all of them.

This mechanism also explains the three regimes without extra assumptions. Pages untouched by the CPU are not yet copy-on-write shared, so the not-resident case never pays for the break. This joins the two halves: breaking copy-on-write is what advances the notifier sequence, whereas a read-only mapping never invalidates, never gets -EBUSY, and never enters the futile retry that c08972f55594 left behind.

4What the split commit cost, and who fixed it

342981fff328 carries neither a Fixes: tag nor Cc: stable@vger.kernel.org, so it was never queued for the stable trees, while c08972f55594 reached -28. An untagged fix does not follow the commit it repairs. That makes this an upstream tagging gap rather than a distro packaging slip, and any series tracking 7.0.y that took the first commit was in the same state.

Four configurations, one reproducer, and only one was slow: the same guest on 7.0.0-28 at sixteen seconds, the same guest on 7.0.0-14 at 18.6 to 20.2 ms, 6.8.0-136 at 22.1 to 26.0, and bare metal on a Proxmox 7.0.14 build at 24.1 to 28.6. The first two rows are the ones that matter: same guest, same ROCm userspace, same container image, same reproducer, only the kernel ABI swapped. Passthrough is not the decisive factor, though it was the leading suspicion here and at AMD.

5What the permanent half is worth

With the second effect gone, the first is measurable end to end for the first time. Four checkpoints, four load paths, page cache controlled per cell — every cell preceded either by drop_caches or by drop_caches plus a full read of the checkpoint — and each cell a fresh process.

The isolated harness is the optimistic one, by 10 to 14 %. A real vllm serve start on the 12B, reading the loader's own Loading weights took line, gives 1.79× warm and 1.52× cold where the harness above says 1.98 and 1.73. The absolute seconds track closely, which makes the harness usable; the ratio does not, because a server start carries work the harness does not. Take the ratios in Figure 2 as the upper end of what the flag is worth.

Three findings in that figure stand out. The 31B cold cell reaches 7.5× because a 21.67 GiB checkpoint against 23.4 GiB of RAM does not merely lose copy-on-write; it pushes the loader into swap. The MoE cell is the counterexample: almost nothing in a checkpoint of 35 743 mostly small tensors is large enough to pay the copy-on-write cost, making the extra host copy pure overhead and leaving the flag slightly worse than doing nothing. And pread, which shipped in safetensors 0.8.0 and is named only in the release notes, is slower than the clone everywhere except the MoE, where it wins.

6The mechanism, in the resident set

Why the penalty is not a constant ratio becomes visible directly rather than by inference once the resident set is sampled during the load. safetensors unmaps a shard the moment the iterator leaves it, so an end-of-loop sample sees an empty process. The first measurement attempt caught nothing and was discarded.

7What is not established

Why the July baselines do not reproduce

Replaying the 2026-07-28 run on -30 without cache control yields 13.87 s on 8B against 36.4 s then, and 126.61 s on 31B against a timeout past 900 s. Cache does not explain this alone. Between sessions, kernel state changed and guest PCIe atomics were enabled. But the 32 MiB reproducer shifted only from 17.0 to 15.3 ms; the per-copy path does not account for the gap, and nothing here explains it. The July table is superseded, not reconciled.

One machine, and it is a guest

All runs used one VFIO guest, gfx1100, on ext4. A bare-metal Ubuntu -28 was closed by someone else: @shineday999 reported an RX 7900 XTX on ROCm 7.2.1 with no virtualisation, same pathology and the same integer window counts. But a stock module as shipped by Canonical on bare metal remains untested; both runs used custom modules or userspace fixes. CDNA is untested, and hosts where RAM dwarfs the checkpoint are unmeasured.

Residency per tensor

The window count is resident_bytes / 2 MiB, established from the reproducer at 32 and 256 MiB and confirmed independently on another board. That an 8 MiB and a 96 MiB tensor each take one window follows from it: they are barely resident when the copy starts, since nothing reads them first. Still, per-tensor residency was never sampled directly to verify that step.

A 17× asymmetry between the two ranks

One run showed roughly 190 ms per ioctl on one rank against 11 ms on the other. Concurrent processes, memory pressure, and torchrun plus RCCL failed to reproduce it; the asymmetry remains unexplained.

8What has changed since