Real-time Stack Export

Description

This function applies to the scenario where the training and inference service processes are suspended to export stack information for fault locating. If you export real-time stack information when the service is not suspended, the following exceptions may occur: signal sending failure, bin file generation timeout, and bin file parsing failure. As a result, stack information cannot be exported. In addition, the stack information of the same suspended process cannot be exported concurrently. Otherwise, the command may fail to be executed.

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product / Atlas A3 inference product	√
Atlas A2 training product / Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	√
Atlas inference product	√
Atlas training product	√

Notes

When exporting real-time stack information, the asys tool searches for the directory where trace logs are stored ($HOME/ascend/atrace/ by default). If there are too many trace log files, the execution of the asys tool may take a long time. Therefore, you are advised to clear trace logs before using the asys tool to export real-time stack information. For details about trace logs, see Viewing Trace Logs in Log Reference.

Command

asys collect -r=stacktrace --remote=pid --all --quiet

Parameters

r : This parameter is mandatory. Set it to stacktrace to export stack information in real time for subsequent fault locating. If this parameter is not set, fault information is collected. However, this parameter cannot be used together with the --remote, --all and --quiet parameters.
After the command is executed successfully, obtain the exported file as prompted.
remote: specifies the IDs of the processes that are suspended. It is mandatory when -r is set to stacktrace.. The ID must be greater than or equal to 2. If the input process ID does not exist, the asys command reports an error and exits.
all: If this parameter is set, the stack information of all threads in the suspended process is exported. This parameter is required when -r is set to stacktrace.
quiet (optional): If this parameter is set, user interaction is disabled during stack information export. If this parameter is not set, user interaction is enabled by default, and you need to confirm whether the signal set for trace processing is enabled on the current server (whether ASCEND_COREDUMP_SIGNAL is set to a value other than none or is not set). This parameter is optional when -r=stacktrace is used.
When real-time stack information is exported, signal 35 needs to be sent to the specified process. If the signal set for trace processing is disabled, the suspended process is stopped and stack information cannot be exported.

For details about the ASCEND_COREDUMP_SIGNAL environment variable and the signal sets for trace processing, see Environment Variables.

Usage Example and Output Description

asys collect -r=stacktrace --remote=468625 --all --quiet

A command output example is as follows:

2025-04-01 02:59:10,344 [ASYS] [INFO]: asys start.
2025-04-01 02:59:10,370 [ASYS] [WARNING]: This command sends signal 35 to the process:468625. If the process is executed to disable signal receiving through the environment variable ASCEND_COREDUMP_SIGNAL=none, the process:468625 will be killed. Are you sure that signal reception is not disabled? (Y/N)
2025-04-01 02:59:11,151 [ASYS] [INFO]: stackcore file path: /home/test/atrace/trace_468625_468625_20250401025858603877/stackcore_event_468625_20250401025910399626/stackcore_tracer_35_468625_atrace_test_20250401025910399934.txt
2025-04-01 02:59:11,151 [ASYS] [INFO]: collect task execute finish.
2025-04-01 02:59:11,151 [ASYS] [INFO]: asys finish.

Parent topic: asys Tool Usage Guide