Documentation

Locating Memory OOM Errors

You can perform the following steps to locate a fault. If the fault persists, contact technical support. After obtaining the logs, click here to contact technical support.

In the preparation phase, you need to collect CANN log files. For details about how to collect CANN log files (including application logs of the host and system logs of the device), see Collecting Memory OOM Error Information. The following uses ${HOME}/err_log_info/ as an example of directory for storing collected logs.

  1. Check the native C/C++ APIs.

    Check whether a native C API malloc or memset is used to allocate the host memory for the application. If yes, use the third-party ASan tool to detect memory errors and optimize the code logic.

  2. Determine the error type based on the API that reports the error in the host application log file ${HOME}/err_log_info/log/[run|debug]/plog/plog-pid_*.log.
    • If the error information indicates that the aclrtMallocHost API provided by CANN is used to allocate the host memory, OOM is caused by the host memory error.

      In this case, search for the keyword _svm_mem_stats_show in the log file and check the memory allocation statistics of components. For details, see Checking Memory Statistics of Each CANN Component. If the memory occupied by each component does not meet the expectation, contact technical support for further locating.

    • If the error information indicates that the aclrtMalloc, aclrtMallocPhysical, hi_mpi_dvpp_malloc, or hi_mpi_dvpp_malloc API provided by CANN is used to allocate the device memory, OOM is caused by the device memory error.

      In this case, search for the keyword _svm_mem_stats_show in the log file and check the memory allocation statistics of the components or framework. For details, see Checking Memory Statistics of Each CANN Component. If the memory occupied by each component does not meet the expectation, contact technical support for further locating.

    • If other error information is displayed, OOM is caused by abnormal device service processes. In this case, search for the keyword svm_mem_stats_show_device_proc_mem in the log file and view the memory statistics of each service process. For details, see Checking the Memory Statistics of the Device Service Processes. If the memory occupied by each service process does not meet the expectation, contact technical support for further locating.