AIV Communication Operator Execution Failure

Symptom

After the AIV mode is enabled on the Atlas A2 training product/Atlas A2 inference product by running export HCCL_OP_EXPANSION_MODE="AIV", HCCL may orchestrate and execute the communication operator in kernel mode. If the operator fails to be executed, information "fault kernel_name=aiv_all_reduce_***" will be recorded in the following key log:

[ERROR] RUNTIME(699131,python):2025-04-24-21:54:17.707.236 [davinci_kernel_task.cc:1268]699131 PrintErrorInfoForDavinciTask:[INIT][DEFAULT]Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=55873, flip_num=2073, fault kernel_name=aiv_all_reduce_***, fault kernel info ext=aiv_all_reduce_910b_bfloat16_t, program id=42, hash=9645272693770703471.

The same task exception is also printed. You can analyze the root cause by referring to Notify Wait Timeout (EI0002), for example, whether a full-scale timeout occurs or whether a rank encounters an exception earlier.

Parent topic: Task Dispatch and Execution Phase