文档

Viewing Logs (Ascend EP Standard Form)

Viewing App Logs

App logs record logs generated during app execution. For example, after the AscendCL app is executed, you can view its logs in $HOME/ascend/log by default, as shown in Figure 1. For details, see Table 1.

  • App logs can be viewed in containers or on physical machines. To ensure that the log tool runs properly in the container, you need to map the path of the dynamic library file libascendalog.so and the path of libc_sec.so which libascendalog.so depends on to the container. The paths are CANN installation path/ascend-toolkit/latest/compiler/lib64/libascendalog.so and CANN installation path/ascend-toolkit/driver/lib64/common/libc_sec.so.
  • Device-side app logs are automatically transferred to the host. If the transfer fails, the logs are directly flushed to the device. If the transfer is successful, the logs are not flushed to the device.
Figure 1 App logs
Table 1 App logs

Log Directory

Description

$HOME/ascend/log/debug/device-id/device-pid_*.log

Debug logs generated when apps are running on the device, including logs of device-side modules such as AI CPU and HCCP.

$HOME/ascend/log/debug/plog/plog-pid_*.log

Debug logs generated when apps are running on the host.

Include logs of compiler components (such as GE, FE, AI CPU, TBE, and HCCL), runtime components (such as AscendCL, GE, and Runtime), and user-mode driver logs.

$HOME/ascend/log/run/device-id/device-pid_*.log

Run logs generated when apps are running on the device.

$HOME/ascend/log/run/plog/plog-pid_*.log

Run logs generated when apps are running on the host.

$HOME/ascend/log/security/device-id/device-pid_*.log

Security logs generated when apps are running on the device.

$HOME/ascend/log/security/plog/plog-pid_*.log

Security logs generated when apps are running on the host.

Note 1: Replace id and pid with the actual device ID and service process ID, respectively. The asterisk (*) in each file name indicates the timestamp when the log file is created.

Note 2: The preceding directories are shared by all apps in a container or on a physical machine. The number of log files increases with the increase of app processes. As a best practice, you should regularly clear the directories to ensure that the service functions properly. You can use the logrotate function provided by the system to segment logs.

Note 3: If the logs are stored in a medium that has a limit on the number of write times, such as the eMMC or flash memory, you are advised to set the log flush path to the path of the memory file system. When starting a service process, you can set the log flush path by using the environment variable ASCEND_PROCESS_LOG_PATH. You can start another permanent process to periodically and quantitatively dump the logs in the memory file system to the eMMC or flash memory.

Note 4: In a container, device-id in the directory is the logical ID.

Other related configurations:

  • Change the path for flushing app logs. You can use the environment variable ASCEND_PROCESS_LOG_PATH to specify a log flush path. If you want the files generated during compilation and running to be flushed to a unified path, you can use ASCEND_WORK_PATH to set the path for storing files exclusively used by a single node.
  • Set the delay for transferring device-side app logs. In the Ascend EP standard form, the device-side slogd process automatically transfers device-side app logs to the host so that users can directly view such logs on the host. Before the service process exits, there is a default delay of 2000 ms for the system to transfer the app logs from the device to the host. The service process exits when the session times out. Logs that are not transferred to the host are directly flushed to the device. You can use the environment variable ASCEND_LOG_DEVICE_FLUSH_TIMEOUT to set a longer delay for transferring device-side app logs to the host.
  • Set the number of log files of each process stored in each app log directory (plog and device-id). By default, only 10 log files transferred to the host can be stored in the plog or device-id log directory per process. The number of saved log files can be set by using the environment variable ASCEND_HOST_LOG_FILE_NUM.
  • Specify the log congestion handling mode: When log congestion occurs or the I/O access performance is poor, the system may lose logs to ensure that the service performance does not deteriorate. To facilitate fault locating, you can use ASCEND_LOG_SYNC_SAVE to prevent log loss in the case of log congestion or poor I/O access performance.

Set the log congestion processing mode.

  • 0: default processing mode. When log congestion occurs or the I/O access performance is poor, the system may lose logs to ensure that the service performance does not deteriorate.
  • 1: Retain logs when log congestion occurs or the I/O access performance is poor. You are advised to set this parameter to 1 to facilitate fault locating.

Viewing System Logs

System logs record system running information. In the Ascend EP standard form, users do not have the permission to log in to the device. Therefore, they need to use the msnpureport tool to transfer system logs from the device to the host. For details, see msnpureport Instructions.

  • It is not supported to view or export device-side system logs (by using the msnpureport tool) in containers.
  • Device-side app logs can be aged. For each type of logs (debug, run, and security), a maximum of 24 device-app-pid directories are allowed by default. The system scans the directories every 15 seconds. If this limit is reached, the earliest directories are automatically deleted.
  • Keep the number of concurrent app processes within 24 to avoid log data loss.