使用mindformer跑官方仓储的llama 7b ， modelarts的notebook 没有问题正常训练，提交训练作业出现报错（都为8卡，完全一样的镜像 mindformer v0.8）：

Traceback (most recent call last): 

  File "/home/ma-user/modelarts/user-job-dir/mindformers/scripts/mf_parallel7/run_mindformer.py", line 340, in <module> 

    main(config_) 

  File "/home/ma-user/modelarts/user-job-dir/mindformers/scripts/mf_parallel7/mindformers/tools/cloud_adapter/cloud_monitor.py", line 43, in wrapper 

    raise exc 

  File "/home/ma-user/modelarts/user-job-dir/mindformers/scripts/mf_parallel7/mindformers/tools/cloud_adapter/cloud_monitor.py", line 33, in wrapper 

    result = run_func(*args, **kwargs) 

  File "/home/ma-user/modelarts/user-job-dir/mindformers/scripts/mf_parallel7/run_mindformer.py", line 84, in main 

    build_context(config) 

  File "/home/ma-user/modelarts/user-job-dir/mindformers/scripts/mf_parallel7/mindformers/core/context/build_context.py", line 42, in build_context 

    local_rank, device_num = init_context(use_parallel=config.use_parallel, 

  File "/home/ma-user/modelarts/user-job-dir/mindformers/scripts/mf_parallel7/mindformers/core/context/build_context.py", line 93, in init_context 

    init() 

  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/communication/management.py", line 171, in init 

    init_hccl() 

RuntimeError: Ascend collective communication initialization failed. 

 

---------------------------------------------------- 

- Framework Error Message: (For framework developers) 

---------------------------------------------------- 

Init hccl graph adapter failed. 

 

---------------------------------------------------- 

- C++ Call Stack: (For framework developers) 

---------------------------------------------------- 

mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_collective_comm_lib.cc:129 Initialize 

mindspore/ccsrc/plugin/device/ascend/hal/hccl_adapter/hccl_adapter.cc:452 InitKernelInfoStore