Comprehensive Check

Description

Involve the stress test, HBM hardware detection, and CPU detection.

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	☓
Atlas inference product	☓
Atlas training product	☓

Notes

You must run the commands related to comprehensive check as the root user on a physical machine.

Command

# AI Core stress test, which may take a long time
asys diagnose -r=stress_detect -d=deviceId --output=path

# HBM detection
asys diagnose -r=hbm_detect -d=deviceId --timeout=num --output=path

# CPU detection
asys diagnose -r=cpu_detect -d=deviceId --timeout=num --output=path

Parameters

r (required): It indicates the detection mode. The values are as follows:
- stress_detect: AI Core stress test
  This function involves operator execution. Therefore, you need to install the operator binary package (named Ascend-cann-*-ops-*.run) in the environment in advance.
  
  AI Core stress test involves voltage adjustment on the device. When the stress test is complete, the voltage can be automatically restored. However, when the stress test exits abnormally, the voltage cannot be automatically restored. In this case, you can manually restore the voltage based on the asys environment configuration. You are advised to obtain the voltage before and after the AI Core stress test to check whether the voltage is abnormal and whether the voltage needs to be restored. For details about how to obtain and restore the voltage, see Environment Configuration.
  Display of the detection result:
  - If no device is specified but there is only one device, only the status of this device is displayed.
  - If the status of all devices is Pass or Warn, Pass - All or Warn - All is displayed.
  - If the status of devices is inconsistent, the status of each device is displayed in sequence. For example, if there are four devices, Pass, Warn, Warn, and Warn are displayed.
  - If the check result is Warn, the check fails. View the plog on the host side (the default path is $HOME/ascend/log/run|debug/plog/plog-pid _*.log), view the log information based on the keyword [ERROR] AML, and locate and rectify the fault based on the error code. The error code starting with 1 indicates that the test case execution or task delivery fails. The error code starting with 2 indicates that the precision comparison fails. The error code starting with 3 indicates a hardware fault.
  - If the detection result is Pass, the detection is successful.
- hbm_detect: HBM detection
  Display of the detection result:
  - If no device is specified but there is only one device, only the status of this device is displayed.
  - If the status of all devices is Pass or Warn, Pass - All or Warn - All is displayed.
  - If the status of devices is inconsistent, the status of each device is displayed in sequence. For example, if there are four devices, Pass, Warn, Warn, and Warn are displayed.
  - If the check result is Warn, the check fails. View the plog on the host (default path: $HOME/ascend/log/run|debug/plog/plog-pid _*.log), search for the keyword [ERROR] AML, and locate and rectify the fault based on the error code. Error codes starting with 1 indicate test case execution failure or task delivery failure. Error codes starting with 4 indicate hardware faults.
  - If the detection result is Pass, the detection is successful. For HBM detection, if the returned value is greater than 0, the value indicates the number of new ECC errors after the detection. This value is used to trigger the reporting and isolation of risk addresses in advance, ensuring normal running of subsequent services.
- cpu_detect: CPU detection
  Display of the detection result:
  - If no device is specified but there is only one device, only the status of this device is displayed.
  - If the status of all devices is Pass, Warn, or Fail, Pass - All, Warn - All, or Fail - All is displayed.
  - If the status of devices is inconsistent, the status of each device is displayed in sequence. For example, if there are four devices, Pass, Warn, Warn, and Fail are displayed.
  - If the detection result is Fail, the hardware faults occur. In this case, contact technical support.
  - If the detection result is Warn, task scheduling problems occur during the detection. View the detailed information in the plog (default path: $HOME/ascend/log/run|debug/plog/plog-pid _*.log) on the host to locate the fault. You can filter logs using the keyword [ERROR] AML.
  - If the detection result is Pass, the detection is successful.
d: specifies the ID of the device to be detected. This parameter is optional. If this parameter is not specified, the detection results of all devices are displayed by default. Pass indicates that the result is normal, and Warn indicates that the result is abnormal.
timeout (optional): It specifies the hardware detection time, in seconds. If this parameter is not specified, the default detection duration is 600 seconds. This field is valid only for HBM detection and CPU detection. Value range for HBM detection: [0, 604800]. The value 0 indicates that only one round of HBM check is performed. The value range for CPU check is [1, 604800].
output (optional): It specifies the directory for storing the detection result file diagnose_result_{time_stamp}.txt. If the command does not contain the output parameter, the output is not flushed to the disk and is displayed only on the terminal screen. If the value of output is empty or invalid, the specified directory does not have the write permission, or the directory fails to be created, the asys tool exits and reports an error.

Usage Example and Output Description

No device is specified, and all devices are normal. The following uses four devices as an example:

asys diagnose -r=stress_detect
 +------------------------+ -----------------------+ 
 | Group of 4 Device      | Diagnostic Result      | 
 +========================+ =======================+ 
 +--- Performance --------+ -----------------------+ 
 | Stress Detect          | Pass - All             | 
 +------------------------+ -----------------------+ 
asys diagnose -r=hbm_detect --timeout=3000
 +------------------------+------------------------+
 | Group of 4 Device      | Diagnostic Result      |
 +========================+========================+
 +--- Hardware -----------+------------------------+
 | HBM Detect             | Pass - All             |
 |                        | (0, 9, 0, 0)           |
 +------------------------+------------------------+
asys diagnose -r=cpu_detect --timeout=3000
 +------------------------+------------------------+
 | Group of 4 Device      | Diagnostic Result      |
 +========================+========================+
 +--- Hardware -----------+------------------------+
 | CPU Detect             | Pass - All             |
 +------------------------+------------------------+

No device is specified, and some devices are normal. The following uses four devices as an example:

asys diagnose -r=stress_detect
 +------------------------+ -----------------------+ 
 | Group of 4 Device      | Diagnostic Result      | 
 +========================+ =======================+ 
 +--- Performance --------+ -----------------------+ 
 | Stress Detect          | Pass, Warn, Pass, Warn | 
 +------------------------+ -----------------------+ 
asys diagnose -r=hbm_detect
 +------------------------+ -----------------------+ 
 | Group of 4 Device      | Diagnostic Result      | 
 +========================+ =======================+ 
 +--- Hardware -----------+ -----------------------+ 
 | HBM Detect             | Pass, Warn, Pass, Warn | 
 |                        | (9, 0, 5, 0)           |
 +------------------------+ -----------------------+ 
asys diagnose -r=cpu_detect
 +------------------------+------------------------+
 | Group of 4 Device      | Diagnostic Result      |
 +========================+========================+
 +--- Hardware -----------+------------------------+
 | CPU Detect             | Pass, Warn, Pass, Fail |
 +------------------------+------------------------+

A device is specified. The following uses device 0 as an example:

asys diagnose -d=0 -r=stress_detect
 +--------------------+------------------------+
 | Device ID: 0       | Diagnostic Result      |
 +====================+========================+
 +--- Performance ----+------------------------+
 | Stress Detect      | Pass                   |
 +--------------------+------------------------+
asys diagnose -d=0 -r=hbm_detect
 +------------------------+------------------------+
 | Device ID: 0           | Diagnostic Result      |
 +========================+========================+
 +--- Hardware -----------+------------------------+
 | HBM Detect             | Pass(9)                |
 +------------------------+------------------------+
asys diagnose -d=0 -r=cpu_detect
 +------------------------+------------------------+
 | Device ID: 0           | Diagnostic Result      |
 +========================+========================+
 +--- Hardware -----------+------------------------+
 | CPU Detect             | Pass                   |
 +------------------------+------------------------+

Parent topic: asys Tool Usage Guide