Errata

"In the GB200 superchip ... 480GB of LPDDR5X at up to ~500 TB/s".

The per superchip CPU memory bandwidth is 512GB/s, not TB/s, see Nvidia documentation (seems like I can't include the URL).

Note from the Author or Editor:
Thanks for catching this. Yes, we should change this.

> Memory pinning speeds up ... because the CUDA driver doesn't have to pin pages on the fly.

This is a bit misleading, because it could imply that in the non-pinned case, the runtime does pin the pages on the fly.

But it's not how it works, in that case it uses a pool of preallocated pinned staging/bounce buffers, and does an intermediate copy - avoiding this intermediate copy is the main reason for the speeduo.

Note from the Author or Editor:
Change sentence to:
Memory pinning speeds up the tensor.to(device) operations because it avoids the intermediate copy through the CUDA driver’s preallocated pinned staging bounce buffers that are typically used when the source is in pageable host memory.

These two paragraphs mentioned buffer `curr` and `next`, but there is no such names in the previous code. If I understand it correctly, the code is using a variable `s = tile % STAGES` to specify which one of the two buffers to use.

Note from the Author or Editor:
confirmed. fixing.

"For example, a 100-trillion-parameter model would require approximately 182 TB of
GPU memory (182 TB = 100 trillion parameters × 16 bits per weight × 8 bits per
byte) to load the model if each parameter is stored in 16-bit (2-byte) precision"

The error is in the formula: It should be 100 trillion parameters × 16 bits per weight ÷ 8 bits per byte (to convert bits to bytes), not × 8 bits per byte.

Note from the Author or Editor:
Confirmed. Good catch!

​> An all-reduce is actually a reduce-scatter followed by a broadcast.

​Suggested Correction: "An all-reduce in the tree algorithm is implemented as a reduce to the root followed by a broadcast."

​In NCCL, the "Tree" algorithm is a pipelined Reduce-then-Broadcast pattern. The term "Reduce-Scatter" specifically refers to an operation where data is partitioned among ranks (typically used in the Ring algorithm). Using "Reduce-Scatter" here is misleading because the Tree algorithm's primary performance characteristic - lower latency but lower bandwidth utilization - stems from the fact that it does not scatter the data, but rather passes the full message through the tree hierarchy.

Note from the Author or Editor:
@Kristen: Please replace that whole section with the following:

Tree and NVLSTree
In the tree algorithm, reductions and broadcasts are done in a tree structure using the spanning tree algorithm. An all-reduce in the tree algorithm is implemented as a reduce to the root followed by a broadcast. A tree can complete an all-reduce in O(log N) steps for N GPUs—as opposed to O(N) for the ring. As such, a tree algorithm provides lower latency for smaller messages. However, it may not fully utilize all links for large messages because not all GPUs transmit all the time. Some GPUs are leaves of the tree and send only one time up the tree, for instance. NCCL’s tree algorithm is optimized and often used for smaller message sizes in which the total time is dominated by the transfer-startup latency. This is known as a latency-dominated workload—in contrast to the ring algorithm’s bandwidth-dominated use case for large messages. Using NVLSTree will enable NVLink SHARP offload.

blocksPerGrid2D and threadsPerBlock2D are not defined.
So no example of dim3.

Note from the Author or Editor:
Will fix. Good catch!

The explanation for why the transpose kernel has bank conflict is backwards. It says that "all threads access tile[constant_row][varying_col]" when the problem is actually because the naive transpose kernel accesses tile[varying_row][constant_col]. Hence the explanation afterward in that paragraph is also incorrect.

Note from the Author or Editor:
fixing. thanks again for catching this!

The explanation for why the transpose kernel has bank conflict is backwards. It says that "all threads access tile[constant_row][varying_col]" when the problem is actually because the naive transpose kernel accesses tile[varying_row][constant_col]. Hence the explanation afterward in that paragraph is also incorrect.

Note from the Author or Editor:
Fixing. Thanks for pointing this out!

The text states that increasing bucket_cap_mb allows operations to launch "sooner." This is technically inaccurate. In PyTorch DDP, a bucket is only synchronized once it is full. Increasing the size from 25MB to 50MB means the system must wait for more gradients to be computed, delaying the initial launch of the NCCL all-reduce. The performance benefit of larger buckets comes from improved network throughput and reduced per-message overhead, not from earlier execution.

Note from the Author or Editor:
good catch! i will update for the next revision.

thanks!!