常见于通信建链阶段,屏显日志报错:EI0008: The CANN versions are inconsistent。
1 2 3 4 5 6 | ERROR : GeOp5_0GEOP::::DoRunAsync Failed Error Message is : EI0008: The CANN versions are inconsistent: tag [HcomAllReduce_6629421139219749105_0], local_version [1.83.T10.0.B206], remote_version [1.83.T10.0.B207] Solution: Install the same CANN version. TraceBack (most recent call last): Call ops_kernel_info_store loadTask fail[FUNC:Distribute][FILE:hccl_task_info.cc][LINE:213] |
集合通信建链时会校验本端与对端Rank的CANN版本一致性,如果CANN版本不一致,HCCL会返回错误并打印错误码EI0008。
确认不同服务器上安装的CANN软件版本是否一致,若不一致则需要安装一致的版本。
grep -r "CannVersion"