[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.603 [hccl_ip_address.cc:76][8874]if name is null. 

[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.612 [adapter_hccp.cc:1562][8874]call trace: hcclRet -> 1 

[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.619 [topoinfo_detect.cc:580][8874]call trace: hcclRet -> 1 

[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.627 [topoinfo_detect.cc:278][8874][Setup][Agent]topo detect generate local rank info failed! rank[0] 

[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.634 [op_base.cc:844][8874][Init][CommRootInfo]errNo[0x0000000005000001] setup topo detect error 

[INFO] HCCP(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.646 [ra_host.c:1793]tid:8874,ra_get_ifnum(1793) : Input parameters: phy_id[0], nic_position:[0], is_all:[0] 

[ERROR] HCCP(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.675 [rs_socket.c:2744]tid:8874,rs_peer_get_ifnum(2744) : param error, g_rs_cb is NULL 

[ERROR] HCCP(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.684 [ra_peer.c:911]tid:8874,ra_peer_get_ifnum(911) : [get][ra_peer_ifnum]rs_peer_get_ifnum failed(-22) 

[ERROR] HCCP(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.709 [ra_host.c:1814]tid:8874,ra_get_ifnum(1814) : [get][ra_ifnum]ra_peer_get_ifnum failed, ret(-22) 

[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.717 [adapter_hccp.cc:1503][8874][Get][IfNum]errNo[0x000000000500000b] ra get if num fail. ret[128303], num[0] 

[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.724 [adapter_hccp.cc:1366][8874]call trace: hcclRet -> 11 

[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.730 [sal.cc:346][8874]call trace: hcclRet -> 11 

[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.739 [op_base.cc:925][8874][InitCommRootInfo]Init failed, return[0x0000000005000001], rankNum[2], rank[0], rootInfo identifier[192.168.0.123%enp69s0_60006_6_1741141678046852], server[0.0.0.0], logicDevId[-1] 

[INFO] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.748 [op_base.cc:1770][8874]Entry-HcclCommDestroy: op_base comm destroy begin 

[INFO] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.772 [op_base.cc:1784][8874][HcclCommDestroy] comm state is IDLE 

[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.781 [hccl_comm.cc:992][8874]errNo[0x0000000005000002] ptr [communicator_] is nullptr, return HCCL_E_PTR 

[INFO] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.806 [op_base.cc:1801][8874]com is not global com 

[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.837 [op_base.cc:1819][8874][HcclCommDestroy] comm is not exist, comm=0xfffc7173e310, group=192.168.0.123%enp69s0_60006_6_1741141678046852, deviceLogicId=6 

[ERROR] HCCL(8473,mindie_llm_backend_connector):2025-03-05-10:28:02.308.853 [op_base.cc:977][8874][Init][CommRootInfoConfig]errNo[0x0000000005000001]HcclCommInitRootInfo failed. 

[2025-03-05 10:28:02.308926] [error] [8874] [hccl_runner.cpp:132] AllGatherHcclRunner:0 HcclCommInitRootInfo fail, error:1, rank:0, rankSize:2 

[2025-03-05 10:28:02.309038] [error] [8874] [comm_pool.h:43] CommPool commCreateFunc fail 

[2025-03-05 10:28:02.309070] [error] [8874] [hccl_runner.cpp:72] AllGatherHcclRunner:0 get hccl comm fail by rank:0 

[2025-03-05 10:28:02.314772] [error] [8874] [all_gather_hccl_runner.cpp:38] hcclComm is null, rank: 0 

[2025-03-05 10:28:02.314804] [error] [8874] [runner.cpp:115] AllGatherHcclRunner_1_1:1 Execute Failed. st: 3 

[2025-03-05 10:28:02.314828] [error] [8874] [graph_runner.cpp:906] WordEmbeddingRunner_1:0  node[1] execute fail, runner name:AllGatherHcclRunner 

[2025-03-05 10:28:02.314845] [error] [8874] [runner.cpp:115] WordEmbeddingRunner_1:1 Execute Failed. st: 3 

[2025-03-05 10:28:02.314858] [error] [8874] [operation_base.cpp:726] WordEmbedding_1 execute WordEmbeddingRunner fail 

[2025-03-05 10:28:02.314869] [error] [8874] [operation_base.cpp:764] WordEmbedding_1 execute fail, error code: 3 

[INFO] TDT(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.374.551 [stub_process_mode_nowin.cpp:60][ProcessQueueForMdc][tid:8872] [TsdClient] it is unnecessary of current mode[0] to grant queue auth to aicpusd 

[INFO] TDT(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.374.566 [stub_process_mode_nowin.cpp:98][OpenInHost][tid:8872] enter into OpenInHost deviceid[7] 

[INFO] TDT(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.374.573 [stub_process_mode_nowin.cpp:102][OpenInHost][tid:8872] host cpu not support 

[INFO] TDT(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.374.580 [process_mode_manager.cpp:185][OpenProcess][tid:8872] [TsdClient][deviceId=7] [sessionId=1] start hccp and computer process success 

[INFO] HCCP(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.374.619 [ra_host.c:214]tid:8872,ra_init(214) : Input parameters: phy_id[7], nic_position:[1] hdc_type:[6] 

[INFO] HCCP(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.374.640 [ra_hdc.c:578]tid:8872,ra_hdc_init(578) : hdc init start! logic id is 7, phy id is 7, hdc_type is 6 

[INFO] HCCP(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.374.966 [ra_hdc.c:613]tid:8872,ra_hdc_init(613) : hdc init OK! phy_id[7] 

[INFO] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.378.222 [adapter_hccp.cc:1536][8872][Get][DeviceIP]hrtGetIfNum success. ifAddrNum[3]. 

[INFO] HCCP(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.378.231 [ra_host.c:1842]tid:8872,ra_get_ifaddrs(1842) : Input parameters: phy_id[7], nic_position:[1], is_all[0], interface num[3] 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.732 [hccl_ip_address.cc:76][8872]if name is null. 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.741 [adapter_hccp.cc:1562][8872]call trace: hcclRet -> 1 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.749 [topoinfo_detect.cc:580][8872]call trace: hcclRet -> 1 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.756 [topoinfo_detect.cc:278][8872][Setup][Agent]topo detect generate local rank info failed! rank[1] 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.765 [op_base.cc:844][8872][Init][CommRootInfo]errNo[0x0000000005000001] setup topo detect error 

[INFO] HCCP(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.779 [ra_host.c:1793]tid:8872,ra_get_ifnum(1793) : Input parameters: phy_id[0], nic_position:[0], is_all:[0] 

[ERROR] HCCP(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.788 [rs_socket.c:2744]tid:8872,rs_peer_get_ifnum(2744) : param error, g_rs_cb is NULL 

[ERROR] HCCP(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.797 [ra_peer.c:911]tid:8872,ra_peer_get_ifnum(911) : [get][ra_peer_ifnum]rs_peer_get_ifnum failed(-22) 

[ERROR] HCCP(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.822 [ra_host.c:1814]tid:8872,ra_get_ifnum(1814) : [get][ra_ifnum]ra_peer_get_ifnum failed, ret(-22) 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.831 [adapter_hccp.cc:1503][8872][Get][IfNum]errNo[0x000000000500000b] ra get if num fail. ret[128303], num[0] 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.838 [adapter_hccp.cc:1366][8872]call trace: hcclRet -> 11 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.844 [sal.cc:346][8872]call trace: hcclRet -> 11 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.855 [op_base.cc:925][8872][InitCommRootInfo]Init failed, return[0x0000000005000001], rankNum[2], rank[1], rootInfo identifier[192.168.0.123%enp69s0_60006_6_1741141678046852], server[0.0.0.0], logicDevId[-1] 

[INFO] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.863 [op_base.cc:1770][8872]Entry-HcclCommDestroy: op_base comm destroy begin 

[INFO] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.877 [op_base.cc:1784][8872][HcclCommDestroy] comm state is IDLE 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.885 [hccl_comm.cc:992][8872]errNo[0x0000000005000002] ptr [communicator_] is nullptr, return HCCL_E_PTR 

[INFO] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.917 [op_base.cc:1801][8872]com is not global com 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.950 [op_base.cc:1819][8872][HcclCommDestroy] comm is not exist, comm=0xfffc9f8f2d60, group=192.168.0.123%enp69s0_60006_6_1741141678046852, deviceLogicId=7 

[ERROR] HCCL(8475,mindie_llm_backend_connector):2025-03-05-10:28:02.382.990 [op_base.cc:977][8872][Init][CommRootInfoConfig]errNo[0x0000000005000001]HcclCommInitRootInfo failed. 

[2025-03-05 10:28:02.383026] [error] [8872] [hccl_runner.cpp:132] AllGatherHcclRunner:0 HcclCommInitRootInfo fail, error:1, rank:1, rankSize:2 

[2025-03-05 10:28:02.383103] [error] [8872] [comm_pool.h:43] CommPool commCreateFunc fail 

[2025-03-05 10:28:02.383115] [error] [8872] [hccl_runner.cpp:72] AllGatherHcclRunner:0 get hccl comm fail by rank:1 

[2025-03-05 10:28:02.387549] [error] [8872] [all_gather_hccl_runner.cpp:38] hcclComm is null, rank: 1 

[2025-03-05 10:28:02.387579] [error] [8872] [runner.cpp:115] AllGatherHcclRunner_1_1:1 Execute Failed. st: 3 

[2025-03-05 10:28:02.387602] [error] [8872] [graph_runner.cpp:906] WordEmbeddingRunner_1:0  node[1] execute fail, runner name:AllGatherHcclRunner 

[2025-03-05 10:28:02.387619] [error] [8872] [runner.cpp:115] WordEmbeddingRunner_1:1 Execute Failed. st: 3 

[2025-03-05 10:28:02.387633] [error] [8872] [operation_base.cpp:726] WordEmbedding_1 execute WordEmbeddingRunner fail 

[2025-03-05 10:28:02.387644] [error] [8872] [operation_base.cpp:764] WordEmbedding_1 execute fail, error code: 3