Error Reported When the Query API Is Called for Asynchronous Copy

Symptom

event is used for synchronous waiting of H2D or D2H asynchronous copy tasks, and aclrtQueryEventStatus is called to confirm task progress. After that, aclrtFreeHost is called to free the host memory before calling aclrtDestroyEvent. As a result, the following error message may be displayed.

[Event] DRV(78295,python):2023-01-10-11:21:48.757.930 [ascend] [curpid: 78295, 78295][drv][common][share_log_read 544][ascend] [ERROR] [devmm] <python:3960,3960> Set free error. (ref_lock=1; ref_free=0; ref_count=3)
[ascend] [ERROR] [devmm] <python:3960,3960> Oper address failed. (va=0x120043200000; ref_flag=0x108; ref_lock=0; ref_free=0; ref_count=3; convert=1; async=0)
[ascend] [ERROR] [devmm] <python:3960,3960> Vaddress can not oper. (cmd=Ox42204d04; cmd_id=Ox4; ret=-22)
[ERROR] DRV(78295,python):2023-01-10-11:21:48,757,946 [ascend][curpid: 78295, 78295][drv][devmm][devmm_ioctl_free_pages 138]<errno:26, 17> Ioctl device error. (ret=17)
[ERROR] DRV(78295,python):2023-01-10-11:21:48,757,951 [ascend][curpid: 78295, 78295][drv][devmm][devmm_virt_heap_free_pages 294]<errno:26, 17> Devmm_iotcl_free failed. (ptr=0x120043200000; heap_type=4025417729)
[ERROR] DRV(78295,python):2023-01-10-11:21:48,757,956 [ascend][curpid: 78295, 78295][drv][devmm][devmm_free_phymem_heap_oper 323]<errno:26, 17> Heap ops failed. (ret=17; va=0x120043200000)
[ERROR] DRV(78295,python):2023-01-10-11:21:48,757,962 [ascend][curpid: 78295, 78295][drv][devmm][devmm_free_nocache_mem_process 992]<errno:26, 17> Free error. (va=Ox120043200000; size=123731968; total=123731968; ret=17)
[ERROR] DRV(78295,python):2023-01-10-11:21:48,757,970 [ascend][curpid: 78295, 78295][drv][devmm][devmm_free_to_normal_heap 704]<errno:26, 17> Virt_heap_free_mem failed. (ret=17; va=Ox120043200000)
[ERROR] RUNTIME(78295,python):2023-01-10-11:21:48.758.013 [npu_driver.cc:1531]78295 HostMemFree:[FINAL][FINAL]report error module_type=1, module_name=EL9999
[ERROR] RUNTIME(78295,python):2023-01-10-11:21:48.758.021 [npu_driver.cc:1531]78295 HostMemFree:[FINAL][FINAL] [drv api] halMemFree failed: drvRetCode=17!
[ERROR] RUNTIME(78295,python):2023-01-10-11:21:48.758.073 [logger.cc:388]78295 HostFree:[FINAL][FINAL]Free host memory failed.
[ERROR] RUNTIME(78295,python):2023-01-10-11:21:48.758.097 [api_c.cc:1077]78295 rtFreeHost:[FINAL][FINAL]ErrCode=507899, desc=[driver error:internal error], InnerCode=0x7020022
[ERROR] RUNTIME(78295,python):2023-01-10-11:21:48.758.102 [error_message_manage.cc:49]78295 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(78295,python):2023-01-10-11:21:48.758.110 [error_message_manage.cc:49]78295 FuncErrorReason:[FINAL][FINAL]rtFreeHost execute failed, reason=[driver error:internal error]
[ERROR] ASCENDCL(78295,python):2023-01-10-11:21:48.758.133 [memory.cpp:242]78295 aclrtFreeHost: [FINAL][FINAL]free host memory failed, runtime result = 507899

Possible Cause

An event record task is delivered after the asynchronous copy task is executed. It is expected that aclrtQueryEventStatus can be used to query whether the event record task is complete to determine whether the asynchronous copy task is complete before calling aclrtFreeHost to free the memory.

However, aclrtQueryEventStatus actually queries that the device does not transparently transmitted data to the host after executing the task. In this case, if the memory is freed before the event is destroyed, a time sequence error occurs.

Solution

To solve this problem, use either of the following solutions:

Solution 1: Call aclrtSynchronizeStream to check whether the task is completed.

Solution 2: Call aclrtDestroyEvent before calling aclrtQueryEventStatus, and then call aclrtFreeHost. This prevents problems caused by time sequence.