Viewing Logs (Ascend EP )
This section describes log file storage paths and main information recorded in log files in
Viewing App Logs
App logs record the logs generated during app running. For example, after an AscendCL app is executed, you can view the app logs in the $HOME/ascend/log directory by default.
App log description:
- App logs can be viewed in containers or on physical machines. To ensure that the log tool runs properly in the container, you need to map the path of the dynamic library file libascendalog.so and the path of libc_sec.so on which libascendalog.so depends to the container. The paths are ${INSTALL_DIR}/compiler/lib64/libascendalog.so and ${INSTALL_DIR}/driver/lib64/common/libc_sec.so, respectively.
Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.
- Device-side app logs are automatically transferred back to the host. If the transfer fails, the logs are directly flushed to disk on the device. If the transfer succeeds, they are not flushed to disk on the device.
- App logs support aging. If the number or size of log files exceeds the threshold, the earliest log directory or file is automatically deleted.
├── debug
│ ├── device-0
│ │ ├── device-2272_20220617234019449.log
│ │ └── device-6564_20220617232348429.log
│ └── plog
│ └── plog-16027_20220618001623878.log
├── run
│ ├── device-0
│ │ ├── device-2272_20220617234019449.log
│ │ └── device-6564_20220617232348429.log
│ └── plog
│ └── plog-16027_20220618001623878.log
└── security
├── device-0
│ ├── device-2272_20220617234019449.log
│ └── device-6564_20220617232348429.log
└── plog
└── plog-16027_20220618001623878.log
Log Directory |
Description |
|---|---|
$HOME/ascend/log/debug/device-id/device-pid_*.log |
Debug logs generated when apps are running on the device, including logs of device-side modules such as AI CPU and HCCP. |
$HOME/ascend/log/debug/plog/plog-pid_*.log |
Debug logs generated when apps are running on the host. Include logs of compiler components (such as GE, FE, AI CPU, TBE, and HCCL), runtime components (such as AscendCL, GE, and Runtime), and user-mode driver logs. |
$HOME/ascend/log/run/device-id/device-pid_*.log |
Run logs generated when apps are running on the device. |
$HOME/ascend/log/run/plog/plog-pid_*.log |
Run logs generated when apps are running on the host. |
$HOME/ascend/log/security/device-id/device-pid_*.log |
Security logs generated when apps are running on the device. |
$HOME/ascend/log/security/plog/plog-pid_*.log |
Security logs generated when apps are running on the host. |
Note 1: Replace id and pid with the actual device ID and service process ID, respectively. The asterisk (*) in each file name indicates the timestamp when the log file is created.
Note 2: The preceding directories are shared by all apps in a container or on a physical machine. The number of log files increases with the increase of app processes. As a best practice, you should regularly clear the directories to ensure that the service functions properly. You can use the logrotate function provided by the system to segment logs.
Note 3: When logs are stored in a medium that has a limit on the number of write operations, such as the eMMC or flash memory, you are advised to set the log output path to the path of the memory file system. Set this log output path via the ASCEND_PROCESS_LOG_PATH environment variable when starting the service process. Spawn an additional resident process to periodically dump logs from the memory file system in batches to eMMC or flash.
Note 4: In a container, device-id in the directory is the logical ID.
Other related configurations:
- Modify the path for flushing app logs. ASCEND_PROCESS_LOG_PATH can be used to specify the path for flushing logs. If you want the files generated during compilation and runtime to be flushed to a unified path, set the storage path for exclusive single-machine files via ASCEND_WORK_PATH.
- Set the delay for transferring device-side app logs. In the
Ascend EP standard form, the device-side slogd process automatically transfers device-side app logs to the host so that users can directly view such logs on the host. Before the service process exits, there is a default delay of 2000 ms for the system to transfer the app logs from the device to the host. The service process exits when the session times out. Logs that are not transferred to the host are directly flushed to the device. You can use the environment variable ASCEND_LOG_DEVICE_FLUSH_TIMEOUT to set a longer delay for transferring device-side app logs to the host. - Setting the number of log files stored for each process in the app log directories (plog and device-id). By default, the plog and device-id directories can store 10 log files transferred from a single process. You can use the environment variable ASCEND_HOST_LOG_FILE_NUM to set the number of log files to store.
- Specify the log congestion handling mode. When log congestion occurs or the I/O access performance is poor, the system may discard logs to prevent service performance deterioration. To facilitate fault locating, you can use ASCEND_LOG_SYNC_SAVE to configure that no log is discarded when log congestion occurs or the I/O access performance is poor.
- Set the log display mode. By default, logs are saved to log files. If you need to print logs, you can configure the environment variable ASCEND_SLOG_PRINT_TO_STDOUT. After the log printing function is enabled, you can also save logs to a specified file in output redirection mode when starting the app process, for example, ./main > log.txt.
Viewing System Logs
System logs record system runtime information. In standard form of
System log description:
- System logs on the device cannot be viewed inside containers, nor can they be exported via the msnpureport tool.
- System logs support aging. If the number or size of log files exceeds the threshold, the earliest log directory or file is automatically deleted.