报错求助RuntimeError: call aclnnAllGatherMatmul failed,RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend
收藏回复举报
报错求助RuntimeError: call aclnnAllGatherMatmul failed,RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend
t('forum.solved') 已解决
发表于2025-01-20 14:49:46
0 查看

具体报错如下所示

RuntimeError: call aclnnAllGatherMatmul failed

RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend

请问怎么解决?

```shell

[rank3]: RuntimeError: call aclnnAllGatherMatmul failed, detail:EI0009: 2025-01-20-06:32:30.807.441 Transport init error. Reason: [Create][Des

tLink]Create Dest error! creakLink para:rank[1]-localUserrank[1]-localIpAddr[172.16.1.36], dst_rank[0]-remoteUserrank[0]-remote_ip_addr[172.16

.1.36]

[rank3]:         Solution: Check other NPUs that are not reporting errors, or check if there are any abnormalities on the other end's NPU.

[rank3]:         TraceBack (most recent call last):

[rank3]:         HcomCreateParamByComm err in HcclAllocComResource, ret = 4 context = (nil).

[rank3]:         HcomCreateParamByComm return nullptr.

[rank3]: [ERROR] 2025-01-20-06:32:30 (PID:262940, Device:3, RankID:3) ERR01100 OPS call acl api failed

[rank2]: RuntimeError: call aclnnAllGatherMatmul failed, detail:EI0009: 2025-01-20-06:32:30.807.454 Transport init error. Reason: [Create][Des

tLink]Create Dest error! creakLink para:rank[0]-localUserrank[0]-localIpAddr[172.16.1.36], dst_rank[1]-remoteUserrank[1]-remote_ip_addr[172.16

.1.36]

[rank2]:         Solution: Check other NPUs that are not reporting errors, or check if there are any abnormalities on the other end's NPU.

[rank2]:         TraceBack (most recent call last):

[rank2]:         HcomCreateParamByComm err in HcclAllocComResource, ret = 4 context = (nil).

[rank2]:         HcomCreateParamByComm return nullptr.

[rank2]: [ERROR] 2025-01-20-06:32:30 (PID:262939, Device:2, RankID:2) ERR01100 OPS call acl api failed

[rank4]: RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:160 OPS function error: HcclBatchSendRecv, error code is 4

[rank4]: [ERROR] 2025-01-20-06:32:31 (PID:262941, Device:4, RankID:4) ERR01100 OPS call acl api failed.

[rank4]: EI0009: 2025-01-20-06:32:31.017.381 Transport init error. Reason: [Create][DestLink]Create Dest error! creakLink para:rank[7]-localUs

errank[4]-localIpAddr[172.16.1.36], dst_rank[0]-remoteUserrank[0]-remote_ip_addr[172.16.1.36]

[rank4]:         Solution: Check other NPUs that are not reporting errors, or check if there are any abnormalities on the other end's NPU.

```

我要发帖子