Viewing Trace Logs

Overview

The trace mechanism records software stack maintenance and diagnosis information in memory during program execution. When the program crashes or the process terminates, this information is flushed to the disk as a file. This avoids frequent log file generation and writing at runtime, which would otherwise degrade the performance. Currently, only the standard Ascend EP supports this function.

Log Description

The default root directory for storing trace logs is $HOME/ascend/atrace/. You can also use the environment variable ASCEND_WORK_PATH to specify the path for storing trace logs, for example, setting export ASCEND_WORK_PATH=/home/test. For details, see Environment Variables.

The path for storing trace logs is $HOME/ascend/atrace/trace_{Process group pid}_{PID of the process that loads the trace dynamic library for the first time}_{Timestamp of loading the trace dynamic library for the first time}/{event_name}_event_{Current process pid}_{Timestamp of generating the directory}/. event_name indicates the event type and the value can be schedule (service process exception, such as operator execution error), stackcore (process crash or exception signal received), or exit (normal exit from destruction).

**Table 1** Description of trace log files
Log Directory	Description
schedule_tracer_ts_{device_id}.txt	When an AI Core error or notify wait timeout occurs, the task schedule returns the maintenance and test information, including the register, hardware buffer, and bitmap, to the host.
stackcore_tracer_{signal}_{tid}_{program_name}_{time}.txt	Lightweight core file recorded when the host service process breaks down, including the stack frame address and base address, needs to be parsed by using the asys tool. For details, see Troubleshooting.
schedule_tracer_{object_name}.txt	Track information reported by modules such as Runtime and HCCL during running, which records the process running process.
schedule_tracer_{object_name}.bin	Track information reported by modules such as AI CPU during running, which records the process running process. The information is stored in binary format. To parse the information, use the asys tool. For details, see Troubleshooting.

You can use the environment variable ASCEND_LOG_DEVICE_FLUSH_TIMEOUT to configure the delay for transferring device-side logs to the host. For details, see Environment Variables.
The environment variable ASCEND_TRACE_RECORD_NUM can be used to control the aging specifications of the schedule_event_{PID of the current process}_{Timestamp when the directory is generated} subdirectory in the $HOME/ascend/atrace/trace_{Process group PID}_{PID of the process that loads the dynamic trace library for the first time}_{Timestamp when the dynamic trace library is loaded for the first time}/ directory. For details, see Environment Variables.

The preceding directories are shared by all apps in a container or on a physical machine. The number of log files increases with the increase of app processes. As a best practice, you should regularly clear the directories to ensure that the service functions properly. You can use the logrotate function provided by the system to segment logs.