Data-parallel scaling and collective bandwidth measurements

Measure step time against the number of ranks with DDP and FSDP and all-reduce bus bandwidth with nccl-tests, then explain the gap from linear scaling.

Not attemptedSelf-directed (nccl-tests, PyTorch DDP and FSDP)Access: freeExtra, for review rotation

Open resource

The exercise happens on Self-directed (nccl-tests, PyTorch DDP and FSDP). Canonical link: https://github.com/NVIDIA/nccl-tests. Link check: link responded (5 Oct 2026). Opened from this app 0 times; opening is not counted as an attempt.

Supports: Needs at least two GPUs for the measurement part; the API part runs on CPU processes with the gloo backend.

Record an attempt

OutcomeAttempted, not solvedSolved with help (hints, editorial or discussion)Solved independentlyReviewed an earlier solution

Date

Minutes (optional)

Language

Hints used

Confidence afterwards

Main mistake

Learning objective this practices (optional)

Notes (approach, bug, what to remember)

Re-attempt on (optional)

A date is suggested from the outcome.

Attempt history

No attempts recorded yet.