NCCL

GPU infrastructure planned for 6-8 kW per node discovers actual power demands of 10-12 kW when enabling higher TDP profiles in production, requiring physical infrastructure renegotiation and topology redesign.

CUDA, driver, NCCL, container runtime, and Kubernetes device plugin version conflicts cause cluster flakiness when not strictly pinned, with uncontrolled upgrades introducing silent failures.