我使用 ray 构建了2机器，4卡的环境：
两台机器情况分别是：
- 机器1：启动的单机8卡docker容器，使用 ASCEND_RT_VISIBLE_DEVICES=2,3，使用其中的 2,3 卡
- 机器2：启动的2卡容器，2卡使用的是机器的6，7卡，容器显示的是 ASCEND_VISIBLE_DEVICES=6,7
torch/torch_npu版本是2.3.1
程序报错：
ascend log 错误日志：
文本信息：
/home/tangzhiyi/ascend/log/debug/plog/plog-152624_20250303023245585.log:[ERROR] DRV(152624,python):2025-03-03-02:32:52.300.199 [ascend][curpid: 152624, 152624][drv][hdc][share_log_read_in_single_module 634]Server create success.(dev_id=0; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=1; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=2; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=3; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=4; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=5; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=6; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=7; fid=0; service_type="service_dump"; pid=196419)
/home/tangzhiyi/ascend/log/debug/plog/plog-152624_20250303023302554.log:[ERROR] DRV(152624,python):2025-03-03-02:33:31.579.452 [ascend][curpid: 152624, 152836][drv][hdc][share_log_read_in_single_module 634]Server destroy success.(dev_id=0; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=1; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=2; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=3; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=4; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=5; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=6; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=7; fid=0; service="service_dump"; pid=196419)
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] DRV(46997,rWrapper.warmup_dist):2025-03-03-02:33:19.079.426 [ascend][curpid: 46997, 153334][drv][hdc][share_log_read_in_single_module 634]Server create success.(dev_id=0; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=1; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=2; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=3; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=4; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=5; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=6; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=7; fid=0; service_type="service_dump"; pid=27871)
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.288 [runtime.cc:5606]153273 GetUserDevIdByDeviceId:[INIT][DEFAULT]get userDevId failed, input deviceId:2 real deviceCnt:8 isSetVisibleDev:1 ASCEND_RT_VISIBLE_DEVICES:[3].
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.320 [api_error.cc:1918]153273 GetDeviceIndexByPhyId:[INIT][DEFAULT]report error module_type=0, module_name=EE9999
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.338 [api_error.cc:1918]153273 GetDeviceIndexByPhyId:[INIT][DEFAULT]get error: phyid:2 realDeviceId:2 is err:0x7010003
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.605 [api_c_device.cc:277]153273 rtGetDeviceIndexByPhyId:[INIT][DEFAULT]ErrCode=107001, desc=[device id error], InnerCode=0x7010003
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.633 [error_message_manage.cc:53]153273 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.662 [error_message_manage.cc:53]153273 FuncErrorReason:[INIT][DEFAULT]rtGetDeviceIndexByPhyId execute failed, reason=[device id error]
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.566 [adapter_rts.cc:379][153273][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[2], devIndex[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.581 [adapter_rts.cc:845][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.593 [adapter_rts.cc:873][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.605 [hccl_communicator_attrs.cc:219][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.616 [hccl_communicator_attrs.cc:198][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.627 [hccl_communicator_attrs.cc:890][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.637 [hccl_communicator_attrs.cc:29][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.647 [hccl_communicator.cc:233][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.659 [hccl_comm.cc:117][153273][HcclComm][Init]errNo[0x000000000500000f] hccl initialize failed
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.671 [op_base.cc:894][153273][InitCommRootInfo]errNo[0x000000000500000f] hcclComm init error
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCP(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.752 [rs_socket.c:2744]tid:153273,rs_peer_get_ifnum(2744) : param error, g_rs_cb is NULL
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCP(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.763 [ra_peer.c:964]tid:153273,ra_peer_get_ifnum(964) : [get][ra_peer_ifnum]rs_peer_get_ifnum failed(-22)
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCP(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.788 [ra_host.c:1930]tid:153273,ra_get_ifnum(1930) : [get][ra_ifnum]ra_peer_get_ifnum failed, ret(-22)
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.805 [adapter_hccp.cc:1492][153273][Get][IfNum]errNo[0x000000000500000b] ra get if num fail. ret[128303], num[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.816 [adapter_hccp.cc:1355][153273]call trace: hcclRet -> 11
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.827 [sal.cc:346][153273]call trace: hcclRet -> 11
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.843 [op_base.cc:922][153273][InitCommRootInfo]Init failed, return[0x000000000500000f], rankNum[4], rank[1], rootInfo identifier[192.168.215.237%bond0_60002_2_1740969205920414], server[0.0.0.0], logicDevId[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.936 [op_base.cc:1816][153273][HcclCommDestroy] comm is not exist, comm=0xffce937a3320, group=192.168.215.237%bond0_60002_2_1740969205920414, deviceLogicId=0
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.593.232 [op_base.cc:974][153273][Init][CommRootInfoConfig]errNo[0x000000000500000f]HcclCommInitRootInfo failed.
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] DRV(46737,rWrapper.warmup_dist):2025-03-03-02:33:19.079.428 [ascend][curpid: 46737, 153335][drv][hdc][share_log_read_in_single_module 634]Server create success.(dev_id=0; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=1; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=2; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=3; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=4; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=5; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=6; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=7; fid=0; service_type="service_dump"; pid=27609)
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.510.706 [runtime.cc:5606]153274 GetUserDevIdByDeviceId:[INIT][DEFAULT]get userDevId failed, input deviceId:3 real deviceCnt:8 isSetVisibleDev:1 ASCEND_RT_VISIBLE_DEVICES:[2].
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.510.730 [api_error.cc:1918]153274 GetDeviceIndexByPhyId:[INIT][DEFAULT]report error module_type=0, module_name=EE9999
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.510.745 [api_error.cc:1918]153274 GetDeviceIndexByPhyId:[INIT][DEFAULT]get error: phyid:3 realDeviceId:3 is err:0x7010003
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.083 [api_c_device.cc:277]153274 rtGetDeviceIndexByPhyId:[INIT][DEFAULT]ErrCode=107001, desc=[device id error], InnerCode=0x7010003
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.107 [error_message_manage.cc:53]153274 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.129 [error_message_manage.cc:53]153274 FuncErrorReason:[INIT][DEFAULT]rtGetDeviceIndexByPhyId execute failed, reason=[device id error]
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.881 [adapter_rts.cc:379][153274][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[3], devIndex[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.893 [adapter_rts.cc:847][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.902 [adapter_rts.cc:873][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.912 [hccl_communicator_attrs.cc:219][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.920 [hccl_communicator_attrs.cc:198][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.929 [hccl_communicator_attrs.cc:890][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.936 [hccl_communicator_attrs.cc:29][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.944 [hccl_communicator.cc:233][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.954 [hccl_comm.cc:117][153274][HcclComm][Init]errNo[0x000000000500000f] hccl initialize failed
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.964 [op_base.cc:894][153274][InitCommRootInfo]errNo[0x000000000500000f] hcclComm init error
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCP(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.038 [rs_socket.c:2744]tid:153274,rs_peer_get_ifnum(2744) : param error, g_rs_cb is NULL
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCP(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.046 [ra_peer.c:964]tid:153274,ra_peer_get_ifnum(964) : [get][ra_peer_ifnum]rs_peer_get_ifnum failed(-22)
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCP(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.056 [ra_host.c:1930]tid:153274,ra_get_ifnum(1930) : [get][ra_ifnum]ra_peer_get_ifnum failed, ret(-22)
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.068 [adapter_hccp.cc:1492][153274][Get][IfNum]errNo[0x000000000500000b] ra get if num fail. ret[128303], num[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.077 [adapter_hccp.cc:1355][153274]call trace: hcclRet -> 11
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.086 [sal.cc:346][153274]call trace: hcclRet -> 11
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.098 [op_base.cc:922][153274][InitCommRootInfo]Init failed, return[0x000000000500000f], rankNum[4], rank[0], rootInfo identifier[192.168.215.237%bond0_60002_2_1740969205920414], server[0.0.0.0], logicDevId[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.177 [op_base.cc:1816][153274][HcclCommDestroy] comm is not exist, comm=0xffceaf820fd0, group=192.168.215.237%bond0_60002_2_1740969205920414, deviceLogicId=0
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.414 [op_base.cc:974][153274][Init][CommRootInfoConfig]errNo[0x000000000500000f]HcclCommInitRootInfo failed.
感觉核心问题是：
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.881 [adapter_rts.cc:379][153274][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[3], devIndex[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.566 [adapter_rts.cc:379][153273][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[2], devIndex[0]
/root/ascend/log/debug/plog/plog-664_20250303023250894.log:[ERROR] HCCL(664,rmup_dist):2025-03-03-02:33:01.812.951 [adapter_rts.cc:379][782][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[6], devIndex[0]
/root/ascend/log/debug/plog/plog-665_20250303023250944.log:[ERROR] HCCL(665,rmup_dist):2025-03-03-02:33:01.773.085 [adapter_rts.cc:379][783][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[7], devIndex[0]
这里我每张卡相当于是一个ray 启动的进程，在当前进程来看，device idx 应该是0，对应的物理卡号是2/3/6/7 应该是没问题的吧？但是报错了，求帮看下是什么原因
初始化代码：
cann版本：8.0.0.beta1
cpu架构：aarch64
硬件：Atlas 800T A2
2机器 ascend log如下：
点赞
收藏
回复
分享
举报
已解决
我使用 ray 构建了2机器，4卡的环境：
两台机器情况分别是：
- 机器1：启动的单机8卡docker容器，使用 ASCEND_RT_VISIBLE_DEVICES=2,3，使用其中的 2,3 卡
- 机器2：启动的2卡容器，2卡使用的是机器的6，7卡，容器显示的是 ASCEND_VISIBLE_DEVICES=6,7
torch/torch_npu版本是2.3.1
程序报错：
ascend log 错误日志：
文本信息：
/home/tangzhiyi/ascend/log/debug/plog/plog-152624_20250303023245585.log:[ERROR] DRV(152624,python):2025-03-03-02:32:52.300.199 [ascend][curpid: 152624, 152624][drv][hdc][share_log_read_in_single_module 634]Server create success.(dev_id=0; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=1; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=2; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=3; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=4; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=5; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=6; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=7; fid=0; service_type="service_dump"; pid=196419)
/home/tangzhiyi/ascend/log/debug/plog/plog-152624_20250303023302554.log:[ERROR] DRV(152624,python):2025-03-03-02:33:31.579.452 [ascend][curpid: 152624, 152836][drv][hdc][share_log_read_in_single_module 634]Server destroy success.(dev_id=0; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=1; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=2; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=3; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=4; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=5; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=6; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=7; fid=0; service="service_dump"; pid=196419)
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] DRV(46997,rWrapper.warmup_dist):2025-03-03-02:33:19.079.426 [ascend][curpid: 46997, 153334][drv][hdc][share_log_read_in_single_module 634]Server create success.(dev_id=0; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=1; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=2; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=3; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=4; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=5; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=6; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=7; fid=0; service_type="service_dump"; pid=27871)
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.288 [runtime.cc:5606]153273 GetUserDevIdByDeviceId:[INIT][DEFAULT]get userDevId failed, input deviceId:2 real deviceCnt:8 isSetVisibleDev:1 ASCEND_RT_VISIBLE_DEVICES:[3].
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.320 [api_error.cc:1918]153273 GetDeviceIndexByPhyId:[INIT][DEFAULT]report error module_type=0, module_name=EE9999
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.338 [api_error.cc:1918]153273 GetDeviceIndexByPhyId:[INIT][DEFAULT]get error: phyid:2 realDeviceId:2 is err:0x7010003
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.605 [api_c_device.cc:277]153273 rtGetDeviceIndexByPhyId:[INIT][DEFAULT]ErrCode=107001, desc=[device id error], InnerCode=0x7010003
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.633 [error_message_manage.cc:53]153273 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.662 [error_message_manage.cc:53]153273 FuncErrorReason:[INIT][DEFAULT]rtGetDeviceIndexByPhyId execute failed, reason=[device id error]
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.566 [adapter_rts.cc:379][153273][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[2], devIndex[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.581 [adapter_rts.cc:845][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.593 [adapter_rts.cc:873][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.605 [hccl_communicator_attrs.cc:219][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.616 [hccl_communicator_attrs.cc:198][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.627 [hccl_communicator_attrs.cc:890][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.637 [hccl_communicator_attrs.cc:29][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.647 [hccl_communicator.cc:233][153273]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.659 [hccl_comm.cc:117][153273][HcclComm][Init]errNo[0x000000000500000f] hccl initialize failed
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.671 [op_base.cc:894][153273][InitCommRootInfo]errNo[0x000000000500000f] hcclComm init error
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCP(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.752 [rs_socket.c:2744]tid:153273,rs_peer_get_ifnum(2744) : param error, g_rs_cb is NULL
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCP(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.763 [ra_peer.c:964]tid:153273,ra_peer_get_ifnum(964) : [get][ra_peer_ifnum]rs_peer_get_ifnum failed(-22)
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCP(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.788 [ra_host.c:1930]tid:153273,ra_get_ifnum(1930) : [get][ra_ifnum]ra_peer_get_ifnum failed, ret(-22)
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.805 [adapter_hccp.cc:1492][153273][Get][IfNum]errNo[0x000000000500000b] ra get if num fail. ret[128303], num[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.816 [adapter_hccp.cc:1355][153273]call trace: hcclRet -> 11
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.827 [sal.cc:346][153273]call trace: hcclRet -> 11
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.843 [op_base.cc:922][153273][InitCommRootInfo]Init failed, return[0x000000000500000f], rankNum[4], rank[1], rootInfo identifier[192.168.215.237%bond0_60002_2_1740969205920414], server[0.0.0.0], logicDevId[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.936 [op_base.cc:1816][153273][HcclCommDestroy] comm is not exist, comm=0xffce937a3320, group=192.168.215.237%bond0_60002_2_1740969205920414, deviceLogicId=0
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.593.232 [op_base.cc:974][153273][Init][CommRootInfoConfig]errNo[0x000000000500000f]HcclCommInitRootInfo failed.
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] DRV(46737,rWrapper.warmup_dist):2025-03-03-02:33:19.079.428 [ascend][curpid: 46737, 153335][drv][hdc][share_log_read_in_single_module 634]Server create success.(dev_id=0; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=1; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=2; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=3; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=4; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=5; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=6; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=7; fid=0; service_type="service_dump"; pid=27609)
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.510.706 [runtime.cc:5606]153274 GetUserDevIdByDeviceId:[INIT][DEFAULT]get userDevId failed, input deviceId:3 real deviceCnt:8 isSetVisibleDev:1 ASCEND_RT_VISIBLE_DEVICES:[2].
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.510.730 [api_error.cc:1918]153274 GetDeviceIndexByPhyId:[INIT][DEFAULT]report error module_type=0, module_name=EE9999
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.510.745 [api_error.cc:1918]153274 GetDeviceIndexByPhyId:[INIT][DEFAULT]get error: phyid:3 realDeviceId:3 is err:0x7010003
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.083 [api_c_device.cc:277]153274 rtGetDeviceIndexByPhyId:[INIT][DEFAULT]ErrCode=107001, desc=[device id error], InnerCode=0x7010003
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.107 [error_message_manage.cc:53]153274 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.129 [error_message_manage.cc:53]153274 FuncErrorReason:[INIT][DEFAULT]rtGetDeviceIndexByPhyId execute failed, reason=[device id error]
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.881 [adapter_rts.cc:379][153274][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[3], devIndex[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.893 [adapter_rts.cc:847][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.902 [adapter_rts.cc:873][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.912 [hccl_communicator_attrs.cc:219][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.920 [hccl_communicator_attrs.cc:198][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.929 [hccl_communicator_attrs.cc:890][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.936 [hccl_communicator_attrs.cc:29][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.944 [hccl_communicator.cc:233][153274]call trace: hcclRet -> 15
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.954 [hccl_comm.cc:117][153274][HcclComm][Init]errNo[0x000000000500000f] hccl initialize failed
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.964 [op_base.cc:894][153274][InitCommRootInfo]errNo[0x000000000500000f] hcclComm init error
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCP(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.038 [rs_socket.c:2744]tid:153274,rs_peer_get_ifnum(2744) : param error, g_rs_cb is NULL
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCP(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.046 [ra_peer.c:964]tid:153274,ra_peer_get_ifnum(964) : [get][ra_peer_ifnum]rs_peer_get_ifnum failed(-22)
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCP(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.056 [ra_host.c:1930]tid:153274,ra_get_ifnum(1930) : [get][ra_ifnum]ra_peer_get_ifnum failed, ret(-22)
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.068 [adapter_hccp.cc:1492][153274][Get][IfNum]errNo[0x000000000500000b] ra get if num fail. ret[128303], num[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.077 [adapter_hccp.cc:1355][153274]call trace: hcclRet -> 11
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.086 [sal.cc:346][153274]call trace: hcclRet -> 11
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.098 [op_base.cc:922][153274][InitCommRootInfo]Init failed, return[0x000000000500000f], rankNum[4], rank[0], rootInfo identifier[192.168.215.237%bond0_60002_2_1740969205920414], server[0.0.0.0], logicDevId[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.177 [op_base.cc:1816][153274][HcclCommDestroy] comm is not exist, comm=0xffceaf820fd0, group=192.168.215.237%bond0_60002_2_1740969205920414, deviceLogicId=0
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.414 [op_base.cc:974][153274][Init][CommRootInfoConfig]errNo[0x000000000500000f]HcclCommInitRootInfo failed.
感觉核心问题是：
/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.881 [adapter_rts.cc:379][153274][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[3], devIndex[0]
/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.566 [adapter_rts.cc:379][153273][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[2], devIndex[0]
/root/ascend/log/debug/plog/plog-664_20250303023250894.log:[ERROR] HCCL(664,rmup_dist):2025-03-03-02:33:01.812.951 [adapter_rts.cc:379][782][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[6], devIndex[0]
/root/ascend/log/debug/plog/plog-665_20250303023250944.log:[ERROR] HCCL(665,rmup_dist):2025-03-03-02:33:01.773.085 [adapter_rts.cc:379][783][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[7], devIndex[0]
这里我每张卡相当于是一个ray 启动的进程，在当前进程来看，device idx 应该是0，对应的物理卡号是2/3/6/7 应该是没问题的吧？但是报错了，求帮看下是什么原因
初始化代码：
def init_process_group(rank: int, world_size: int): """init process group.""" from lmdeploy.pytorch.devices import get_device_manager # 获取当前设备类型 device_type = get_device_manager().current_context().device_type # 根据设备类型选择合适的后端 if device_type == 'ascend': backend = 'hccl' else: backend = 'nccl' DIST_TIMEOUT = timedelta(days=35600) dist.init_process_group(backend=backend, rank=rank, world_size=world_size, timeout=DIST_TIMEOUT) assert dist.is_initialized()cann版本：8.0.0.beta1
cpu架构：aarch64
硬件：Atlas 800T A2
2机器 ascend log如下：