ERROR] 2024-01-27 12:48:48,083 [mindformers/tools/cloud_adapter/cloud_monitor.py:43] wrapper: Traceback (most recent call last):
File "/home/ma-user/work/files/mindformers/scripts/mf_parallel1/mindformers/core/context/build_context.py", line 109, in init_context
init()
File "/home/ma-user/anaconda3/envs/mindspore_py39/lib/python3.9/site-packages/mindspore/communication/management.py", line 181, in init
init_cluster()
RuntimeError: Maybe you are trying to call 'mindspore.communication.init()' without using 'mpirun', which will make MindSpore load several environment variables and check their validation. Please use 'mpirun' to launch this process to fix this issue, or refer to this link if you want to run distributed training without using 'mpirun': https://www.mindspore.cn/tutorials/experts/zh-CN/master/parallel/train_gpu.html
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/distributed/collective/collective_manager.cc:154 Initialize
During handling of the above exception, another exception occurred:
ERROR] 2024-01-27 12:48:48,083 [mindformers/tools/cloud_adapter/cloud_monitor.py:43] wrapper: Traceback (most recent call last):
File "/home/ma-user/work/files/mindformers/scripts/mf_parallel1/mindformers/core/context/build_context.py", line 109, in init_context
init()
File "/home/ma-user/anaconda3/envs/mindspore_py39/lib/python3.9/site-packages/mindspore/communication/management.py", line 181, in init
init_cluster()
RuntimeError: Maybe you are trying to call 'mindspore.communication.init()' without using 'mpirun', which will make MindSpore load several environment variables and check their validation. Please use 'mpirun' to launch this process to fix this issue, or refer to this link if you want to run distributed training without using 'mpirun': https://www.mindspore.cn/tutorials/experts/zh-CN/master/parallel/train_gpu.html
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/distributed/collective/collective_manager.cc:154 Initialize
During handling of the above exception, another exception occurred: