torch.distributed

Initialization

表1 Initialization

序号

API名称

是否支持

1

torch.distributed.Backend

2

torch.distributed.get_backend

3

torch.distributed.get_rank

4

torch.distributed.get_world_size

5

torch.distributed.init_process_group

6

torch.distributed.is_initialized

7

torch.distributed.is_mpi_available

8

torch.distributed.is_nccl_available

Groups

表2 Groups

序号

API名称

是否支持

1

torch.distributed.new_group

Point-to-point communication

表3 Point-to-point communication

序号

API名称

是否支持

1

torch.distributed.irecv

2

torch.distributed.isend

3

torch.distributed.recv

4

torch.distributed.send

Synchronous and asynchronous collective operations

表4 Synchronous and asynchronous collective operations

序号

API名称

是否支持

1

is_completed

2

wait

Collective functions

表5 Collective functions

序号

API名称

是否支持

1

torch.distributed.all_gather

2

torch.distributed.all_reduce

3

torch.distributed.barrier

4

torch.distributed.broadcast

5

torch.distributed.gather

6

torch.distributed.reduce

7

torch.distributed.reduce_op

8

torch.distributed.ReduceOp

9

torch.distributed.scatter

Multi-GPU collective functions

表6 Multi-GPU collective functions

序号

API名称

是否支持

1

torch.distributed.all_gather_multigpu

2

torch.distributed.all_reduce_multigpu

3

torch.distributed.broadcast_multigpu

4

torch.distributed.reduce_multigpu

Launch utility

表7 Launch utility

序号

API名称

是否支持

1

torch.distributed.launch

Spawn utility

表8 Spawn utility

序号

API名称

是否支持

1

torch.multiprocessing.spawn