NCCLinSGLang

文档摘要

NCCL in SGLang Before we discuss NVIDIA Collective Communication Library (NCCL) and communication in SGLang, we introduce the parallelism and general communication primitives. Parallelism In this section, we introcue the parallelism - TP, DP, PP. Generally we assume the communication data size should be TP > DP > PP.