Fault Information Collection
Description
Collect fault information such as software and hardware information and logs without service re-run.
Applicability
Product |
Supported |
|---|---|
Atlas 350 Accelerator Card |
√ |
√ |
|
√ |
|
√ |
|
√ |
|
√ |
Command
asys collect --task_dir=path1 --tar="True" --output=path2
Parameters
- task_dir (optional): It specifies the directory for collecting operator compilation files (including .o and .json files) and dump files (including GE dump graphs, TF Adapter dump graphs, and exception dump files). If task_dir is not specified or no file is collected from the directory specified by task_dir, the asys tool automatically collects the files from the environment.
Automatic collection is affected by environment variables. Therefore, when you run the asys command to collect files, the environment variable values must be the same as those used during service running. Otherwise, the collected information may be incorrect. The involved environment variables are ASCEND_PROCESS_LOG_PATH, NPU_COLLECT_PATH, DUMP_GRAPH_PATH, ASCEND_WORK_PATH, ASCEND_CACHE_PATH and ASCEND_CUSTOM_OPP_PATH. For details about the environment variables and their restrictions, see Environment Variables.
- tar (optional): It controls whether to compress the result output directory of the asys tool into a *.tar.gz file. If this parameter is set to T or True, the result output directory is compressed into a *.tar.gz file and the original directory is not retained. If this parameter is set to F or False, the directory is not compressed into a *.tar.gz file. By default, the directory is not compressed. The parameter value is case insensitive.
- output (optional): Its value is used as the prefix of the result output directory of the asys tool. That is, the final output directory is {output}/asys_output_timestamp. If the command does not contain the output parameter, the output is stored in the command execution directory. If the value of output is empty or invalid, the specified directory does not have the write permission, or the directory fails to be created, the asys tool exits and reports an error.
Usage Example and Output Description
asys collect --task_dir=$HOME/cache_path --tar="True" --output=$HOME/dfx_info
After the command is executed, the fault information files in {output}/asys_output_timestamp are as follows:
├── asys_output_timestamp
├── software_info.txt // Installation package version, environment variables, dependent software, and system information.
├── hardware_info.txt // Collecting host and device hardware information. The host information includes the kernel version, CPU model, memory usage, and disk usage. The device information includes the number of devices and the number of AI CPUs.
├── status_info.txt // Collecting device information, including the chip model, CPU usage, and AI Core usage.
├── health_result.txt // Collecting device health information, including fault codes and fault information.
└── dfx
├── bbox // Black box information on the device.
├── data-dump // Generating L0 exception dump files when an AI Core error occurs in L0.
├── graph // Dump graph information, including the dump graphs of the GE and TF Adapter. L0 exception dump does not collect this information.
├── ops // Operator compilation information, including the .o and .json operator compilation files and custom operator configuration.
├── stackcore // Information about the core file generated when a core dump is triggered by an error.
├── atrace // Trace flush information, including the plaintext file parsed from the trace binary file.
└── log
├── device
│ ├──dev-os-{id}
│ ├── firmware // Logs generated by the firmware.
│ ├── slogd // Maintenance and test logs of log-related processes.
│ ├── application // The non-event application logs generated by the service process.
│ └── system // Logs generated by resident process.
└── host
├── message // messages or syslogs.
├── install // Logs about the history installation of the package.
↔ cann // Application logs
└── driver // Driver logs on the host.
You can accordingly customize the version information of the third-party dependent software collected in the software_info.txt file. In the ascend_system_advisor/conf/dependent_package.csv file in the asys tool directory, add or delete configuration items. Each line corresponds to a configuration item. Use commas (,) to separate dependency item names and query commands. There should be no space after each comma. The following is an example:
make,make --version cmake,cmake --version unzip,unzip -v zlib1g,dpkg -l zlib1g| grep zlib1g| grep ii zlib1g-dev,dpkg -l zlib1g-dev| grep zlib1g-dev| grep ii libsqlite3-dev,dpkg -l libsqlite3-dev| grep libsqlite3-dev| grep ii openssl,dpkg -l openssl| grep openssl| grep ii libssl-dev,dpkg -l libssl-dev| grep libssl-dev| grep ii libffi-dev,dpkg -l libffi-dev| grep libffi-dev| grep ii