【问题描述】：
模型多机训练任务启动初始化失败
【告警信息】:
EI0009: 2025-01-06-11:45:24.276.468 Transport init error. Reason: [Create][DestLink]Create Dest error! creakLink para:rank[0]-localUserrank[0]-localIpAddr[10.240.32.158], dst_rank[1]-remoteUserrank[8]-remote_ip_addr[10.240.33.78]
Solution: Check other NPUs that are not reporting errors, or check if there are any abnormalities on the other end's NPU.
【产品及版本】:
CANN 8.0.RC1
【分析过程】：
1. 训练日志报错，错误码EI0009。
2. 查看plog日志，显示HCCP 错误
[ERROR] HCCP(2210,python3):2025-01-06-10:04:09.261.455 [ra_hdc.c:876]tid:11543,ra_hdc_qp_create_with_attrs(876) : [create][ra_hdc_qp_with_attrs]ra hdc message process failed ret(-12) phy_id(0)
[ERROR] HCCP(2210,python3):2025-01-06-10:04:09.261.464 [ra_host.c:1390]tid:11543,ra_qp_create_with_attrs(1390) : [create][ra_qp_with_attrs]create qp fail, ret(-12) phy_id(0)
3. 进一步查看device日志，RoCE用户态驱动报错，显示HCCP内存初始化失败
ERROR] ROCE(23559,hccp_service.bin):2025-01-06-10:04:10.520.336 [hns_roce_hal.c:82]hns_roce_hal_alloc_buf(82) : hns_roce_hal_alloc_buf mmap fail. start addr 0xffffffffffffffff length 0x400000
[ERROR] ROCE(23559,hccp_service.bin):2025-01-06-10:04:10.520.362 [hns_roce_u_buf.c:210]roce_init_mem_pool(210) : hns init pool failed, hal_alloc_buf failed ret = -12
[ERROR] HCCP(23559,hccp_service.bin):2025-01-06-10:04:10.520.371 [rs_rdma.c:1453]tid:23569,rs_init_mem_pool(1453) : rs_roce_init_mem_pool failed, ret=-12, chip_id=0
[ERROR] HCCP(23559,hccp_service.bin):2025-01-06-10:04:10.520.379 [rs_rdma.c:1655]tid:23569,rs_alloc_qpcb_with_attrs(1655) : init mem pool failed ret -12
[ERROR] HCCP(23559,hccp_service.bin):2025-01-06-10:04:10.520.395 [rs_rdma.c:1714]tid:23569,rs_qp_create_with_attrs(1714) : alloc mem for qp_cb fail, ret:-12
[ERROR] HCCP(23559,hccp_service.bin):2025-01-06-10:04:10.520.408 [ra_adp.c:553]tid:23569,ra_rs_qp_create_with_attrs(553) : qp create failed ret[-12].
【根因】：
CANN8.0.RC1版本质量问题，在主机内存页大小如果不是4k的场景下会默认开启RDMA lite特性，HCCP通信会申请2MB大页映射内存，如果申请不到则会导致HCCP通信失败。
【解决方案】：
升级CANN至8.0.RC2以及之后版本。
【相关知识】：
使用CANN 8.0.RC2之后的版本，增加了RDMA lite特性开关，可以设置 HCCL_RDMA_PCIE_DIRECT_POST_NOSTRICT = FALSE来关闭该特性。
https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha003/apiref/envref/envref_07_0093.html
点赞
收藏
回复
分享
举报
【问题描述】：
模型多机训练任务启动初始化失败
【告警信息】:
EI0009: 2025-01-06-11:45:24.276.468 Transport init error. Reason: [Create][DestLink]Create Dest error! creakLink para:rank[0]-localUserrank[0]-localIpAddr[10.240.32.158], dst_rank[1]-remoteUserrank[8]-remote_ip_addr[10.240.33.78]
Solution: Check other NPUs that are not reporting errors, or check if there are any abnormalities on the other end's NPU.
【产品及版本】:
CANN 8.0.RC1
【分析过程】：
1. 训练日志报错，错误码EI0009。
2. 查看plog日志，显示HCCP 错误
[ERROR] HCCP(2210,python3):2025-01-06-10:04:09.261.455 [ra_hdc.c:876]tid:11543,ra_hdc_qp_create_with_attrs(876) : [create][ra_hdc_qp_with_attrs]ra hdc message process failed ret(-12) phy_id(0)
[ERROR] HCCP(2210,python3):2025-01-06-10:04:09.261.464 [ra_host.c:1390]tid:11543,ra_qp_create_with_attrs(1390) : [create][ra_qp_with_attrs]create qp fail, ret(-12) phy_id(0)
3. 进一步查看device日志，RoCE用户态驱动报错，显示HCCP内存初始化失败
ERROR] ROCE(23559,hccp_service.bin):2025-01-06-10:04:10.520.336 [hns_roce_hal.c:82]hns_roce_hal_alloc_buf(82) : hns_roce_hal_alloc_buf mmap fail. start addr 0xffffffffffffffff length 0x400000
[ERROR] ROCE(23559,hccp_service.bin):2025-01-06-10:04:10.520.362 [hns_roce_u_buf.c:210]roce_init_mem_pool(210) : hns init pool failed, hal_alloc_buf failed ret = -12
[ERROR] HCCP(23559,hccp_service.bin):2025-01-06-10:04:10.520.371 [rs_rdma.c:1453]tid:23569,rs_init_mem_pool(1453) : rs_roce_init_mem_pool failed, ret=-12, chip_id=0
[ERROR] HCCP(23559,hccp_service.bin):2025-01-06-10:04:10.520.379 [rs_rdma.c:1655]tid:23569,rs_alloc_qpcb_with_attrs(1655) : init mem pool failed ret -12
[ERROR] HCCP(23559,hccp_service.bin):2025-01-06-10:04:10.520.395 [rs_rdma.c:1714]tid:23569,rs_qp_create_with_attrs(1714) : alloc mem for qp_cb fail, ret:-12
[ERROR] HCCP(23559,hccp_service.bin):2025-01-06-10:04:10.520.408 [ra_adp.c:553]tid:23569,ra_rs_qp_create_with_attrs(553) : qp create failed ret[-12].
【根因】：
CANN8.0.RC1版本质量问题，在主机内存页大小如果不是4k的场景下会默认开启RDMA lite特性，HCCP通信会申请2MB大页映射内存，如果申请不到则会导致HCCP通信失败。
【解决方案】：
升级CANN至8.0.RC2以及之后版本。
【相关知识】：
使用CANN 8.0.RC2之后的版本，增加了RDMA lite特性开关，可以设置 HCCL_RDMA_PCIE_DIRECT_POST_NOSTRICT = FALSE来关闭该特性。
https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha003/apiref/envref/envref_07_0093.html