Service Re-run And Fault Information Collection

Description

Collect fault information such as software and hardware information and logs after services are re-run.

Applicability

Notes

• By default, the function of collecting operator compilation files, GE dump graphs, and TF Adapter dump graphs is enabled during service re-run.
• When the asys launch command is executed, the NPU_COLLECT_PATH, ASCEND_PROCESS_LOG_PATH, ASCEND_HOST_LOG_FILE_NUM and ASCEND_WORK_PATH environment variables are automatically enabled to temporarily store the collected information. After the asys launch command is executed, these environment variables are automatically disabled. If you manually set these environment variables before running the asys launch command, the environment variables set by the user will be overwritten and do not take effect. If the user task script to be executed contains the NPU_COLLECT_PATH, ASCEND_PROCESS_LOG_PATH, ASCEND_HOST_LOG_FILE_NUM and ASCEND_WORK_PATH environment variables, the environment variables set by the asys tool may be overwritten and do not take effect. As a result, the corresponding information cannot be collected. For details about environment variables and their restrictions, see Environment Variables.
• The asys launch command starts the subprocess to execute the service command. If you stop the launch command, the service subprocess may not exit. In this case, you need to stop the service subprocess.

Command

asys launch --task="sh ../app_run.sh" --tar="True" --output=path

Parameters

• task: (mandatory) command for resuming a service. You need to enter a complete command, for example, "sh ../app_run.sh". sh indicates the command for executing the task, and ../app_run.sh indicates the task script to be executed.

  The tool cannot be directly executed in the background inside the original execution script. For example, the original test case is started by running the sh cmd.sh command. However, the python3 test.py & command is executed in the cmd.sh command in the background. This task is not supported because the end point cannot be detected.

• tar (optional): It controls whether to compress the result output directory of the asys tool into a *.tar.gz file. If this parameter is set to T or True, the result output directory is compressed into a *.tar.gz file and the original directory is not retained. If this parameter is set to F or False, the directory is not compressed into a *.tar.gz file. By default, the directory is not compressed. The parameter value is case insensitive.
• output (optional): Its value is used as the prefix of the result output directory of the asys tool. That is, the final output directory is {output}/asys_output_timestamp. If the command does not contain the output parameter, the output is stored in the command execution directory. If the value of output is empty or invalid, the specified directory does not have the write permission, or the directory fails to be created, the asys tool exits and reports an error.

Usage Example and Output Description

1. (Optional) Modify the configuration items related to service re-run. If the configuration items are not modified, the default configurations are used.

   The default configuration is as follows. You can modify the parameters in the ascend_system_advisor/asys/conf/asys.ini file in the asys tool directory to enable or disable the operator build file and dump graph collection functions.

   [launch]
   graph = TRUE // Whether to collect graph information. The options are as follows: TRUE indicates that data is collected, and FALSE indicates that data is not collected. If this parameter is set to FALSE, the configurations of dump_ge_graph and dump_graph_level do not take effect.
   ops = TRUE                      // Specifying whether to collect operator compilation information. The value range is as follows: TRUE: collect; FALSE: not collect.
   dump_ge_graph = 2               // Specifying the content of the dump graph. The value is 2. A dump of basic version and does not contain data such as weights. The corresponding environment variable is DUMP_GE_GRAPH.
   dump_graph_level = 3            // Specifying the number of the dump graph. The value is 3. Graphs generated in the last dump phase correspond to the environment variable DUMP_GRAPH_LEVEL.
   log_level = INFO                // Indicating the global log level for application logs and the log level for each module, corresponding to the environment variable ASCEND_GLOBAL_LOG_LEVEL.
   log_event_enable = TRUE         // Determining whether to enable the event log for application logs. The value range is as follows: TRUE: enabled; FALSE: disabled. The corresponding environment variable is ASCEND_GLOBAL_EVENT_ENABLE.
   log_print_to_stdout = FALSE // Whether to print logs on the terminal screen. The options are as follows: TRUE: enabled; FALSE: disabled, corresponding to the environment variable ASCEND_SLOG_PRINT_TO_STDOUT

   By default, the environment variables upon asys startup use the values configured in this file. If these environment variables are set to other values in the script of the re-run task, the environment variables will be overwritten, and the values set in the re-run task script will be used. Therefore, the collected maintenance and test information may not meet the location requirements.

2. Run the command to re-run the service and collect fault information.

   asys launch --task="sh ../run.sh" --tar="True" --output=$HOME/dfx_info

   After the command is executed, the fault information files in {output}/asys_output_timestamp are as follows:

   ├── asys_output_timestamp         
      ├── software_info.txt            // Installation package version, environment variables, dependent software, and system information.
      ├── hardware_info.txt            // Collecting host and device hardware information. The host information includes the kernel version, CPU model, memory usage, and disk usage. The device information includes the number of devices and the number of AI CPUs. 
      ├── status_info.txt              // Collecting device information, including the chip model, CPU usage, and AI Core usage.
      ├── health_result.txt            // Collecting device health information, including fault codes and fault information.
      └── dfx              
          ├── bbox                     // Black box information on the device.
          ├── data-dump                // Dump file generated when an AI Core error occurs.
          ├── graph                    // Dump graph information, including the dump graphs of the GE and TF Adapter.
          ├── ops                      // Operator compilation information, including .o and .json operator compilation files, operator compilation process information, and custom operator configuration.
          ├── stackcore                // Information about the core file generated when a core dump is triggered by an error.
          ├── atrace                  // Trace flush information, including the plaintext file parsed from the trace binary file.
          └── log          
              ├── device       
              │     ├──dev-os-{id}
              │           ├── firmware      // Logs generated by the firmware.
              │           ├── slogd         // Maintenance and test logs of log-related processes.
              │           ├── application   // The non-event application logs generated by the service process.
              │           └── system        // Logs generated by resident process.
              └── host     
                   ├── message         // messages or syslogs.
                   ├── install         // Logs about the history installation of the package.
                   ├── cann            // Application logs
                   ├── driver          // Driver logs on the host.
                   ├── screen.txt      // Log printed on the terminal screen. (If the content is empty, redirection may be set in the application task.)
                   └── user_cmd        // Command used by a user to execute a task.

   You can accordingly customize the version information of the third-party dependent software collected in the software_info.txt file. In the ascend_system_advisor/conf/dependent_package.csv file in the asys tool directory, add or delete configuration items. Each line corresponds to a configuration item. Use commas (,) to separate dependency item names and query commands. There should be no space after each comma. The following is an example:

   make,make --version
   cmake,cmake --version
   unzip,unzip -v
   zlib1g,dpkg -l zlib1g| grep zlib1g| grep ii
   zlib1g-dev,dpkg -l zlib1g-dev| grep zlib1g-dev| grep ii
   libsqlite3-dev,dpkg -l libsqlite3-dev| grep libsqlite3-dev| grep ii
   openssl,dpkg -l openssl| grep openssl| grep ii
   libssl-dev,dpkg -l libssl-dev| grep libssl-dev| grep ii
   libffi-dev,dpkg -l libffi-dev| grep libffi-dev| grep ii