Locating Memory OOM Errors

You can perform the following steps to locate a fault. If the fault persists, contact technical support. After obtaining the logs, click here to contact technical support.

In the preparation phase, you need to collect CANN log files. For details about how to collect CANN log files (including app logs and device system logs), see Collecting Memory OOM Error Information. The following uses ${HOME}/err_log_info/ as an example of directory for storing collected logs.

The locating phase is as follows:

1. Check the native APIs of the C language.

   Check whether a standard C API malloc or memset is used to allocate the host memory for the application. If so, use the third-party ASan tool to detect memory errors and optimize the code logic.

2. Check the CANN memory allocation.
   In the ${HOME}/err_log_info/ log/[run|debug]/plog/plog-pid _*.log file, find the logs generated around the time when the OOM problem occurs and determine the problem type based on the error interface.

   • If the error information indicates that the aclrtMallocHost API provided by CANN is used to allocate the host memory, OOM is caused by the host memory error.

     In this case, search for the keyword mem_stats in the log to view the memory application statistics of each component. For details about how to analyze the log, see Checking Memory Statistics of Each CANN Component. If the memory usage of each component does not meet the expectation, contact technical support to locate the fault.

   • If the error information indicates that the aclrtMalloc, aclrtMallocPhysical, and hi_mpi_dvpp_malloc APIs provided by CANN are used to allocate the device memory, OOM is caused by the device memory error.

     In this scenario, you can search for the keyword mem_stats in the log to view the memory application statistics of each component or framework. For details about the log analysis method, see Checking Memory Statistics of Each CANN Component. If the memory usage of each component does not meet the expectation, contact technical support to locate the fault.

   • Other error messages are generally caused by OOM caused by abnormal device service processes. You can search for the keyword DEV_PROC_MEM in the log to view the memory statistics of each service process. For details about how to analyze the log, see Checking the Memory Statistics of the Device Service Processes. If the memory occupied by the service process does not meet the expectation, contact technical support to locate the fault.