[WARNING] ME(27927:140065362257728,MainProcess):2023-11-06-11:06:53.129.088 [mindspore/run_check/_check_version.py:102] MindSpore version 2.2.0 and cuda version 11.8.89 does not match, CUDA version [['10.1', '11.1', '11.6']] are supported by MindSpore officially. Please refer to the installation guide for version matching information: https://www.mindspore.cn/install. 

[WARNING] DISTRIBUTED(27927,7f638229a740,python3):2023-11-06-11:06:57.283.712 [mindspore/ccsrc/distributed/collective/collective_manager.cc:220] CreateCommunicationGroup] Start to create communication group: nccl_world_group [const vector]{0} 

[WARNING] DISTRIBUTED(27927,7f638229a740,python3):2023-11-06-11:06:57.334.086 [mindspore/ccsrc/distributed/collective/collective_manager.cc:278] CreateCommunicationGroup] Begin initialize communication group on the device side: nccl_world_group 

[ERROR] DISTRIBUTED(27927,7f62a2055700,python3):2023-11-06-11:06:57.379.827 [mindspore/ccsrc/distributed/collective/collective_manager.cc:86] operator()] Failed to execute function asynchronously 

[ERROR] DISTRIBUTED(27927,7f638229a740,python3):2023-11-06-11:06:57.379.867 [mindspore/ccsrc/distributed/collective/collective_manager.cc:285] CreateCommunicationGroup] Failed to create comm group on device side for nccl_world_group 

[WARNING] DISTRIBUTED(27927,7f638229a740,python3):2023-11-06-11:06:57.379.886 [mindspore/ccsrc/distributed/collective/collective_manager.cc:287] CreateCommunicationGroup] End initialize communication group on the device side: nccl_world_group 

[ERROR] DISTRIBUTED(27927,7f638229a740,python3):2023-11-06-11:06:57.379.905 [mindspore/ccsrc/distributed/collective/collective_manager.cc:179] Initialize] Failed to create group nccl_world_group 

Traceback (most recent call last): 

  File "net.py", line 26, in <module> 

    init() 

  File "/usr/local/lib/python3.7/dist-packages/mindspore/communication/management.py", line 177, in init 

    init_cluster()

+-----------------------------------------------------------------------------+ 

| NVIDIA-SMI 470.141.03   Driver Version: 470.141.03   CUDA Version: 11.4     | 

|-------------------------------+----------------------+----------------------+ 

| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC | 

| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. | 

|                               |                      |               MIG M. | 

|===============================+======================+======================| 

|   0  Tesla V100-PCIE...  Off  | 00000000:5A:00.0 Off |                    0 | 

| N/A   43C    P0    39W / 250W |      0MiB / 16160MiB |      0%      Default | 

|                               |                      |                  N/A | 

+-------------------------------+----------------------+----------------------+ 

|   1  Tesla V100-PCIE...  Off  | 00000000:62:00.0 Off |                    0 | 

| N/A   44C    P0    36W / 250W |      0MiB / 16160MiB |      0%      Default | 

|                               |                      |                  N/A | 

+-------------------------------+----------------------+----------------------+ 

|   2  Tesla V100-PCIE...  Off  | 00000000:66:00.0 Off |                    0 | 

| N/A   42C    P0    36W / 250W |      0MiB / 16160MiB |      0%      Default | 

|                               |                      |                  N/A | 

+-------------------------------+----------------------+----------------------+ 

|   3  Tesla V100-PCIE...  Off  | 00000000:B5:00.0 Off |                    0 | 

| N/A   38C    P0    38W / 250W |      0MiB / 16160MiB |      0%      Default | 

|                               |                      |                  N/A | 

+-------------------------------+----------------------+----------------------+ 

|   4  Tesla V100-PCIE...  Off  | 00000000:BD:00.0 Off |                    0 | 

| N/A   40C    P0    38W / 250W |      0MiB / 16160MiB |      2%      Default | 

|                               |                      |                  N/A | 

+-------------------------------+----------------------+----------------------+ 

                                                                                

+-----------------------------------------------------------------------------+ 

| Processes:                                                                  | 

|  GPU   GI   CI        PID   Type   Process name                  GPU Memory | 

|        ID   ID                                                   Usage      | 

|=============================================================================| 

|  No running processes found                                                 | 

+-----------------------------------------------------------------------------+