Swept with rccl_allgather_truth.py across channels and transports on two Radeons; verified against a cross-rank variant and an injected fault
An upstream issue reports deterministic silent corruption during a collective on two gfx1100 cards. On this pair of gfx1100 cards, the corruption does not appear. That clean run proves almost nothing on its own. It gains value only through three concrete checks that do not depend on running the test more times.
Issue ROCm#6565 reports that
all_gather silently returns corrupted data on two W7800s. The upstream report traces the fault to an init-time
8-byte copy whose payload is missing from device memory when synchronisation returns. It reproduces deterministically for them on RCCL 2.27.7 and 2.28.3. On this box running 2.30.4, no corruption appears across
135 cold communicator initialisations in eight configurations.
Three things make that negative result informative. First, a sweep across the dimensions their mechanism names. The reported fault affects only the first of several concurrent init copies. Because this host initialises two channels by default, the sweep scales channel count from one to sixteen and switches the transport from shared memory to sockets. Second, a stated contrast between the two testbeds. The machines differ in platform, kernel, and RCCL version, but agree on PCIe atomics and IOMMU, so the obvious explanations are named rather than left for the reader to guess at. Third, a bound on what a clean run from someone else's script can say.
That third one turned out to be the interesting one.
The upstream reproducer computes corruption checks across all ranks but tallies failures only on rank 0;
consequently, corruption visible only on rank 1 would print ALL CORRECT.
The negative result was therefore narrower than originally reported.
That limitation was disclosed the same night. A cross-rank variant was then written, and all eight arms were re-run through it.
Finally, rather than merely debating the script logic, the blind spot was demonstrated on this
hardware by injecting a one-sided fault.
"It works here" is the least useful sentence on a bug tracker. It can mean the reporter is mistaken, the reporter ran a configuration omitted here, or the test harness cannot observe the failure mode. Turning a negative result into actionable evidence requires answering three questions before posting: could this configuration trigger the bug, what differs between the two machines, and what blind spots exist in the test tool.
This first question makes the parameter sweep essential. The failure mechanism is specific rather than generic: the first-issued copy on channel 0 fails on both ranks, whereas the copy on channel 1 lands. The reporter found that a single-device allocate → memset → 8-byte async copy → synchronise → read loop never reproduced the fault in 200 tries. Concurrent multi-device initialisation is therefore a necessary ingredient. A benchmark that never issues concurrent initialisation copies tests nothing relevant.
torchrun, so each count is that many cold communicator initialisations rather
than repeats inside one process — their failure is deterministic, ours could have been
probabilistic, and a single pass is not evidence of absence. Tallies are parsed back out of the
committed logs rather than read from a summary file, so a stale summary cannot agree with prose the
logs contradict. The two panels are the same eight arms at the same counts, before and after the
verdict was made cross-rank.
The channel sweep gives the negative result its evidentiary weight. By default, this host initialises two channels, matching the exact topology where the first copy fails while the second lands. Forcing one, four, eight, and sixteen channels alters the number of concurrent initialisation copies, while disabling shared memory forces transport across sockets. Across all configurations, execution remains completely clean.
A negative result isolates one specific cell within a larger parameter space. Its utility depends entirely on identifying that cell with precision. Attributes that agree between systems matter as much as those that differ: if this host lacked PCIe atomics or ran without an IOMMU, maintainers could dismiss the observation immediately.
benchmarks/rccl-6565/README.md, so the article cannot drift from the document behind
it. The atomics row is the axis this repository is
otherwise known for: since 2026-08-23 this guest advertises PCIe AtomicOp completer support, so it is
not an atomics-starved cell and does not differ from theirs there. The version and platform axes are
not separated by any of this, and the two runs that would separate them are named
in the source rather than hand-waved.
The initial sweep executed the reporter's script verbatim from the issue tracker, without retyping, citing its md5 checksum for independent verification. Running the unmodified upstream script made the negative result credible, but it also imported the script's hidden defect.
bad_list = [...] # computed on every rank bad_into = [...] # computed on every rank if rank == 0: # and only rank 0 counts, or prints if bad_list or bad_into: fail += 1
When corruption occurs exclusively on rank 1, the script detects the bad data but discards the result, reporting
ALL CORRECT regardless. Since every pass among the 135 runs relied on that tally, the negative
result established only that there was "no corruption observed on rank 0". That limitation was documented in the repository
ten hours after the upstream comment was posted.
Modifying the reporter's script directly was not an acceptable fix. Altering the script, even to insert a cross-rank reduction, destroys its status as an impartial test. The original file was therefore preserved alongside its cryptographic hash, an independent variant was implemented, and both were evaluated in parallel.
The variant was then evaluated across the identical eight arms and sample sizes, recording 135 of 135 clean executions verified across both ranks. Both sweeps are validated to ensure they cover the same configurations at matching sample counts, preventing any silent divergence.
grep -c, which tallies
lines rather than occurrences. Because both ranks frequently write output on a single line, the harness misreported 6 of 20 runs as one-sided
when both ranks had in fact reported. No measurement was compromised—since the script performs an internal all-reduce—but this mirrors the exact failure mode under
investigation. A check that miscounts is worse than no check. Counting token occurrences resolved the error.
The verification plan included a second witness that bypassed PyTorch by employing the reporter's alternative
reproducer. The rccl-tests suite compiled cleanly against the container toolchain, producing confirmed gfx1100 code
objects for both benchmark and verification binaries. However, both logged
all_gather runs aborted on rank 1 inside the data-initialisation kernel with
HIP failure: 'invalid device function'. The abort occurs identically whether the
verification pass is enabled or disabled, clearing the verification kernels of blame. Subsequent arms in the test script never executed, and the test tree remained on a deprecated branch against RCCL 2.30.4.
Rather than discarding this abort, the raw unfiltered log was committed to record the constraint. An
earlier run used a wrapper to parse console output, but its regex matched nothing, producing spurious
FAILED verdicts for every collective—including workloads that the PyTorch harness passed 135 times over. That corrupted log
was deliberately discarded—because a wrong verdict line is worse to leave lying around than no
log—and the raw execution trace replaced it.
This host differs on RCCL version and platform, so a clean run isolates neither factor. Two tests would separate them, but neither is cheap here. The reporter can test 2.30.4 in minutes from source. Deploying 2.28.3 here requires a 73 GB image against 40 GB free, destroying active container state. Both missing runs are documented in source so successors need not rediscover them.
The clean run bounds the defect as not universal to gfx1100 under RCCL, and absent at 2.30.4 on this host across all tested channels and transports. It sets no bound on the reporter's own configuration at all, and it cannot: their failure is deterministic on their machine.
Only one reproducer ran. It is the reporter's own primary one, which is the right one to have; an independent second path would have been better, and the rccl-tests attempt did not produce one here.
rccl-tests builds and "then every collective aborts on rank 1".
In fact, benchmarks/rccl-6565/logs/stage2b-raw.log holds two invocations, both running
all_gather_perf; nothing under that directory carries
reduce_scatter_perf or all_reduce_perf output.
While scripts/stage2b_rccltests.sh planned six arms, the run stopped after the second.
Script intent is not committed evidence, which is the distinction this article is about.
Separately, the sweep was billed as one "that could have exposed it"; what it covers is the
dimensions their mechanism names, and the injection demonstrates the checker rather than the fault.
Everything here. All scripts, variants, execution logs, injection traces, and hardware fingerprints are committed in
benchmarks/rccl-6565/.
Document figures are verified directly from those raw logs by
verify_doc_figures.py,
which confirms identical arm coverage and sample counts between sweeps.
Reported to ROCm#6565, whose reporter is @BoJl4apa. Why this machine is not atomics-starved is a different article.