Viewing Logs (
Ascend EP
)
This section describes log file storage paths and main information recorded in log files in
Viewing App Logs
App logs record the logs generated during app running. For example, after an AscendCL app is executed, you can view the app logs in the $HOME/ascend/log directory by default.
- App logs can be viewed in containers or on physical machines. To ensure that the log tool runs properly in the container, you need to map the path of the dynamic library file libascendalog.so and the path of libc_sec.so which libascendalog.so depends on to the container. The paths are ${INSTALL_DIR}/compiler/lib64/libascendalog.so and ${INSTALL_DIR}/driver/lib64/common/libc_sec.so, respectively.
Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.
- Device-side app logs are automatically transferred to the host. If the transfer fails, the logs are directly flushed to the device. If the transfer is successful, the logs are not flushed to the device.
- App logs support aging. If the number or size of log files exceeds the threshold, the earliest log directory or file is automatically deleted.
├── debug
│ ├── device-0
│ │ ├── device-2272_20220617234019449.log
│ │ └── device-6564_20220617232348429.log
│ └── plog
│ └── plog-16027_20220618001623878.log
├── run
│ ├── device-0
│ │ ├── device-2272_20220617234019449.log
│ │ └── device-6564_20220617232348429.log
│ └── plog
│ └── plog-16027_20220618001623878.log
└── security
├── device-0
│ ├── device-2272_20220617234019449.log
│ └── device-6564_20220617232348429.log
└── plog
└── plog-16027_20220618001623878.log
|
Log Directory |
Description |
|---|---|
|
$HOME/ascend/log/debug/device-id/device-pid_*.log |
Debug logs generated when apps are running on the device, including logs of device-side modules such as AI CPU and HCCP. |
|
$HOME/ascend/log/debug/plog/plog-pid_*.log |
Debug logs generated when apps are running on the host. Include logs of compiler components (such as GE, FE, AI CPU, TBE, and HCCL), runtime components (such as AscendCL, GE, and Runtime), and user-mode driver logs. |
|
$HOME/ascend/log/run/device-id/device-pid_*.log |
Run logs generated when apps are running on the device. |
|
$HOME/ascend/log/run/plog/plog-pid_*.log |
Run logs generated when apps are running on the host. |
|
$HOME/ascend/log/security/device-id/device-pid_*.log |
Security logs generated when apps are running on the device. |
|
$HOME/ascend/log/security/plog/plog-pid_*.log |
Security logs generated when apps are running on the host. |
Note 1: Replace id and pid with the actual device ID and service process ID, respectively. The asterisk (*) in each file name indicates the timestamp when the log file is created.
Note 2: The preceding directories are shared by all apps in a container or on a physical machine. The number of log files increases with the increase of app processes. As a best practice, you should regularly clear the directories to ensure that the service functions properly. You can use the logrotate function provided by the system to segment logs.
Note 3: If the logs are stored in a medium that has a limit on the number of write times, such as the eMMC or flash memory, you are advised to set the log flush path to the path of the memory file system. When starting a service process, you can set the log flush path by using the environment variable ASCEND_PROCESS_LOG_PATH. You can start another permanent process to periodically and quantitatively dump the logs in the memory file system to the eMMC or flash memory.
Note 4: In a container, device-id in the directory is the logical ID.
Other related configurations:
- Change the path for flushing app logs. You can use the environment variable ASCEND_PROCESS_LOG_PATH to specify a log flush path. If you want the files generated during compilation and running to be flushed to a unified path, you can use ASCEND_WORK_PATH to set the path for storing files exclusively used by a single node.
- Set the delay for transferring device-side app logs. In the
Ascend EP standard form, the device-side slogd process automatically transfers device-side app logs to the host so that users can directly view such logs on the host. Before the service process exits, there is a default delay of 2000 ms for the system to transfer the app logs from the device to the host. The service process exits when the session times out. Logs that are not transferred to the host are directly flushed to the device. You can use the environment variable ASCEND_LOG_DEVICE_FLUSH_TIMEOUT to set a longer delay for transferring device-side app logs to the host. - Set the number of log files of each process stored in each app log directory (plog and device-id). By default, only 10 log files transferred to the host can be stored in the plog or device-id log directory per process. The number of saved log files can be set by using the environment variable ASCEND_HOST_LOG_FILE_NUM.
- Specify the log congestion handling mode. When log congestion occurs or the I/O access performance is poor, the system may discard logs to prevent service performance deterioration. To facilitate fault locating, you can use ASCEND_LOG_SYNC_SAVE to configure that no log is discarded when log congestion occurs or the I/O access performance is poor.
- Set the log display mode. By default, logs are saved to log files. If you need to print logs, you can configure the environment variable ASCEND_SLOG_PRINT_TO_STDOUT. After the log printing function is enabled, you can also save logs to a specified file in output redirection mode when starting the app process. For example: ./main > log.txt.
Viewing System Logs
System logs record system running information. In standard form of
- It is not supported to view or export device-side system logs (by using the msnpureport tool) in containers.
- System logs support aging. If the number or size of log files exceeds the threshold, the earliest log directory or file is automatically deleted.