Obtaining Fault IDs from Device Logs and Rectifying RAS Hardware Faults

RAS hardware faults refer to faults related to hardware reliability, availability, and serviceability. You can find the solution in the fault code manual of the corresponding version based on the event_id value (RAS hardware fault code) in the log.

Figure 1 Troubleshooting process

1. On the host server, use the msnpureport tool to export device-side system logs and other maintenance and test information, including slogs, syslogs, event logs, and black box logs.

   Run the msnpureport tool command in a directory (for example, /var/log/npu/report) on which you have the read, write, and execute permissions.

   The following is an example of the msnpureport tool command:

   msnpureport -f

   The exported logs and files are stored in the /var/log/npu/report directory by default.

2. In the event logs (slog/dev-os-id /run/event/event_*.log) collected from 1, find the log information generated around the time when the fault occurs and the device, and check whether the keyword event_id exists in the log. If the keyword does not exist, go to 3. If the keyword exists, click Health Management Fault Definition to obtain the manual of the target version and view the solution.

   If the time in the slog is not around the time when the error occurs, the old log may have been overwritten or deleted. In this case, event_id related to the error cannot be found.

   You can run the npu-smi command to query the health status of a specified chip. If an RAS fault occurs, you can query the event IDs of the last eight faults, which can be used as a reference for fault locating.

   The following is an example of the npu-smi command (id indicates the device ID, and chip_id indicates the chip ID. You can run the npu-smi info command to obtain the device ID and chip ID first):

   npu-smi info -t health -i id -c chip_id

   The following is an example of the query result.

   

3. In the black box logs collected from 1, find the black box logs of the corresponding device near the fault occurrence time in the hisi_logs/device-id /*/bbox directory , and check whether the logs contain the keyword Hardware Error. If not, the hardware fault is not identified. If yes, the hardware fault is unknown. In this case, contact technical support for further fault locating.. After obtaining the logs, click here to contact technical support.