在MindIE RC3的镜像里安装FlagEmbedding和DeepSpeed，然后执行脚本，报错如下：

驱动：24.1.rc3

ASCEND_RT_VISIBLE_DEVICES=4 sh finetune_12333_emb_content.sh  

[2025-05-09 17:33:19,638] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to npu (auto detect) 

[2025-05-09 17:33:21,738] [INFO] [comm.py:669:init_distributed] cdb=None 

[2025-05-09 17:33:21,738] [INFO] [comm.py:700:init_distributed] Initializing TorchBackend in DeepSpeed with backend hccl 

Traceback (most recent call last): 

  File "<frozen runpy>", line 198, in _run_module_as_main 

  File "<frozen runpy>", line 88, in _run_code 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/FlagEmbedding/finetune/embedder/encoder_only/m3/__main__.py", line 27, in <module> 

    main() 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/FlagEmbedding/finetune/embedder/encoder_only/m3/__main__.py", line 13, in main 

    model_args, data_args, training_args = parser.parse_args_into_dataclasses() 

                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/transformers/hf_argparser.py", line 358, in parse_args_into_dataclasses 

    obj = dtype(**inputs) 

          ^^^^^^^^^^^^^^^ 

  File "<string>", line 143, in __init__ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/transformers/training_args.py", line 1761, in __post_init__ 

    self.device 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/transformers/training_args.py", line 2297, in device 

    return self._setup_devices 

           ^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/transformers/utils/generic.py", line 67, in __get__ 

    cached = self.fget(obj) 

             ^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/transformers/training_args.py", line 2224, in _setup_devices 

    self.distributed_state = PartialState(**accelerator_state_kwargs) 

                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/accelerate/state.py", line 203, in __init__ 

    dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs) 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 702, in init_distributed 

    cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size) 

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/deepspeed/comm/torch.py", line 118, in __init__ 

    self.init_process_group(backend, timeout, init_method, rank, world_size) 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/deepspeed/comm/torch.py", line 148, in init_process_group 

    torch.distributed.init_process_group(backend, 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 74, in wrapper 

    func_return = func(*args, **kwargs) 

                  ^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1148, in init_process_group 

    default_pg, _ = _new_process_group_helper( 

                    ^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1309, in _new_process_group_helper 

    backend_class = creator_fn(dist_backend_opts, pg_options) 

                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/torch_npu/__init__.py", line 198, in <lambda> 

    _new_process_group_hccl_helper(dist_backend_opts, pg_options), extended_api=True, devices=["npu"]) 

    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/python3.11.10/lib/python3.11/site-packages/torch_npu/__init__.py", line 193, in _new_process_group_hccl_helper 

    return torch_npu._C._distributed_c10d.ProcessGroupHCCL(store, group_rank, group_size, pg_options) 

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

RuntimeError: ProcessGroupHCCL:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:628 NPU function error: c10_npu::acl::AclrtSetOpWaitTimeout(kOpWaitTimeout), error code is 107002 

[Error]: The context is empty. 

        Check whether acl.rt.set_context or acl.rt.set_device is called. 

EE1001: [PID: 26686] 2025-05-09-17:33:21.753.910 The argument is invalid.Reason: rtSetOpWaitTimeOut execute failed, reason=[context pointer null] 

        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship. 

        TraceBack (most recent call last): 

        ctx is NULL![FUNC:SetOpWaitTimeOut][FILE:api_impl.cc][LINE:5128] 

        The argument is invalid.Reason: rtSetOpWaitTimeOut execute failed, reason=[context pointer null] 

        [Call][Rts]call rts api [rtSetOpWaitTimeOut] failed, retCode is 107002[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] 

        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:5315] 

        The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]