[object Object][object Object]
  • To enable the diagnosis function, it is recommended that the number of servers in a cluster be fewer than or equal to 128 (1024 cards) due to the limitation of the maximum number of processes (1024 by default) in Linux. If the number of servers exceeds the upper limit, run the [object Object] command to adjust the upper limit of file descriptors.
  • Do not use pipe commands when using MindCluster Ascend FaultDiag commands. Otherwise, user IP address acquisition and log audit may be affected.
[object Object]
  • MindCluster Ascend FaultDiag provides fault diagnosis capabilities exclusively for training and inference tasks on servers with full-card configurations. In other scenarios, root cause localization may be inaccurate or may fail.
  • MindCluster Ascend FaultDiag supports only IPv4 addresses.
[object Object]
  • Synchronize the system time of each training or inference server. If the system time is inconsistent, the analysis result may be inaccurate.
  • Synchronize the system time of the host on each training or inference server with that of the device. If the system time is inconsistent, the analysis result may be inaccurate.
  • If a container is used to execute training or inference tasks, synchronize the system time of the host with that of the training or inference container. If the system time is inconsistent, the analysis result may be inaccurate.
[object Object]

Table 1 Software versions corresponding to logs

[object Object]undefined