(Optional) Customizing Fault Entities

You can customize fault entities to extend the fault types supported by MindCluster Ascend FaultDiag by adding, querying, or deleting fault entities. User-added faults are saved in the ${HOME}/.ascend_faultdiag/custom-ascend-kg-config.json file. When you perform log cleaning and dumping and fault diagnosis, MindCluster Ascend FaultDiag automatically loads the user-defined fault files and the fault files already supported by MindCluster Ascend FaultDiag from the corresponding paths.

Procedure

1. Add or modify a custom fault entity through a JSON file.

   ascend-fd entity --update updated_entity.json

   The following is an example of the command output, indicating that the operation is successful.

   Updated entity successfully.

   The following is an example of a JSON file. This example cannot be used directly. You need to modify the information about the custom fault based on the actual situation. A JSON file can contain a maximum of 1,000 custom fault information entries. Any excess entries will not be saved to the system. For details about the parameters in the file, see Table 1.

   {
       "41001": {      #Fault code. You need to customize the fault code based on the actual situation. It cannot be the same as the fault codes already supported by MindCluster Ascend FaultDiag.
           "attribute.class": "Software",
           "attribute.component": "AI Framework",
           "attribute.module": "Compiler",
           "attribute.cause_zh": "抽象类型合并失败",
           "attribute.description_zh": "对函数输出求梯度时，抽象类型不匹配，导致抽象类型合并失败。",
           "attribute.suggestion_zh": [
                  "1. 检查求梯度的函数的输出类型与sens_param的类型是否相同，如果不相同，修改为相同类型；",
                  "2. 自动求导报错Type Join Failed"
              ],
           "attribute.cause_en": "Abstract type merge failed",
           "attribute.description_en": "When computing the gradient of a function output, the abstract types do not match, leading to a failure in abstract type merging.",
           "attribute.suggestion_en": [
                  "1. Check whether the output type of the gradient calculation function matches the type of sens_param. If they do not match, modify them to be of the same type.",
                  "2. Automatic differentiation reports an error: Type Join Failed."
              ],
           "attribute.error_case": [
               "grad = ops.GradOperation(sens_param=True)",
               "# The output type of test_net is tuple(Tensor, Tensor)",
               "def test_net(a, b):",
               "    return a, b"
                 ],
           "attribute.fixed_case": [
               "grad = ops.GradOperation(sens_param=True)",
               "# The output type of test_net is tuple(Tensor, Tensor)",
               "def test_net(a, b):",
               "    return a, b"
               ],
           "rule": [
               {
                   "dst_code": "20106"
               }
           ],
           "source_file": "TrainLog",
           "regex.in": [
               "Abstract type", "cannot join with"
               ]
       },
       "41002": {                #Fault code. Users need to customize the fault code based on actual conditions. It must not be the same as the fault codes already supported by MindCluster Ascend FaultDiag.
           "attribute.class": "",
           "attribute.component": "",
           "attribute.module": "",
           "attribute.cause_zh": "",
           "attribute.description_zh": "",
           "attribute.suggestion_zh": "",
           "attribute.cause_en": "",
           "attribute.description_en": "",
           "attribute.suggestion_en": "",
           "attribute.error_case": "",
           "attribute.fixed_case": "",
           "rule": [
               {
                   "dst_code": "20107"
               }
           ],
           "source_file": "CANN_Plog",
           "regex.in": [
                   "tsd client wait response fail"
               ]
       }
   ...
   }

   In the JSON file example, 41001 and 41002 are user-defined fault codes. The value length must be 1 to 50 characters. English letters, digits, English symbols, underscores (_), and hyphens (-) are supported. The fault code cannot be the same as the fault codes already supported by MindCluster Ascend FaultDiag.

   Table 1 Parameter description

   Parameter Name	Value Type	Parameter Description	Mandatory	Value Description
   attribute.class	String	Fault category	Mandatory	The value length ranges from 1 to 50 characters. English letters, digits, English symbols, and spaces are supported.
   attribute.component	String	Fault component	Mandatory
   attribute.module	String	Fault module	Mandatory
   attribute.cause_zh	String	Fault cause (Chinese)	Mandatory	The value length ranges from 1 to 200 characters. English letters, digits, English symbols, Chinese characters, Chinese symbols, and spaces are supported.
   attribute.cause_en	String	Fault cause (English)	Optional	The value length ranges from 1 to 200 characters. English letters, digits, English symbols, and spaces are supported.
   attribute.description_zh	String	Fault description (Chinese)	Mandatory	Supports strings or lists. A string represents the entire message and can contain line breaks. A list represents the entire message with each element as one line of information. String: The value length ranges from 1 to 2,000 characters. English letters, digits, English symbols, Chinese characters, Chinese symbols, spaces, and "\n" are supported. List: The value length of each string in the list ranges from 1 to 200 characters. English letters, digits, English symbols, Chinese characters, Chinese symbols, and spaces are supported.
   attribute.description_en	String	Fault description (English)	Optional
   attribute.suggestion_zh	String	Suggestion (Chinese)	Mandatory
   attribute.suggestion_en	String	Suggestion (English)	Optional
   attribute.error_case	String	Error example	Optional
   attribute.fixed_case	String	Fixed example	Optional
   rule	List	Fault chain, storing all next-level fault entities triggered by this fault	Optional	The list contains the following fields. dst_code: Mandatory, indicating the fault code of the next-level fault entity triggered by this fault. This fault code must be a fault code supported by MindCluster Ascend FaultDiag or a user-defined fault code. expression: Optional, indicating the fault trigger constraint. This is a reserved field. The value length ranges from 1 to 200 characters. English letters, digits, English symbols, and spaces are supported.
   source_file	String	Fault log file	Mandatory	Log file name corresponding to each log file type. You can customize the file type or use the default supported file types. Multiple configurations are supported, separated by "|" (for example, "TrainLog|CANN_Plog"). A maximum of 10 can be configured. The value length of each string ranges from 1 to 50 characters. English letters, digits, English symbols, and spaces are supported. The default supported log file types are as follows (for the storage directory corresponding to the file name, see Table 1 Log file list). TrainLog: Training and inference console log. CANN_Plog: Host-side application log. CANN_Device: Device-side application log. NPU_OS: System log on the Device-side Control CPU and EVENT-level system log on the Device-side Control CPU. NPU_Device: System log on the Device-side non-Control CPU. NPU_History: Black box log, NPU chip kernel log, Device-side OS basic information, and Device-side on-chip memory log. OS: Host-side operating system log file. OS-dmesg: Host-side kernel message file. OS-vmcore-dmesg: Host-side kernel message log file saved during a system crash. OS-sysmon: Host-side system monitoring file. NodeDLog: AI server log. DL_DevicePlugin: Super node device log, Ascend Device Plugin component log. DL_Volcano_Scheduler: Volcano volcano-scheduler component log. DL_Volcano_Controller: Volcano volcano-controller component log. DL_Docker_Runtime: Ascend Docker Runtime component log. DL_Npu_Exporter: NPU Exporter component log. MindIE: MindIE component log. CANN_Amct: AMCT component log.
   regex.in	String	Fault keyword	Mandatory	Supports first-level lists and second-level lists. First-level list Each element is a string. The value length ranges from 1 to 200 characters. English letters, digits, English symbols, Chinese characters, Chinese symbols, and spaces are supported. Each keyword in the list must satisfy the existence check and conform to the sequential relationship. Second-level list Each sub-list satisfies the value constraints of the first-level list. The judgment rule within each sub-list is the same as that of the first-level list. The relationship between sub-lists is OR, meaning that only the keywords of one sub-list need to be satisfied.
   When adding a custom fault entity, all mandatory fields must exist in the JSON file and comply with the relevant value requirements. When modifying a custom fault entity, only the relevant value requirements need to be met.

2. View user-defined custom fault entity information. You can query fault information by fault code. If no fault code is specified, all custom fault entity information will be queried.

   ascend-fd entity --show entity_code_1 entity_code_2

3. (Optional) Delete the custom fault entity information corresponding to the specified fault code.

   ascend-fd entity --delete entity_code_1 entity_code_2

4. (Optional) Verify the custom-ascend-kg-config.json file. If you have directly modified the custom fault entity information in this file, you can run the following command to verify the integrity and availability of the modified file.

   Directly modifying the custom-ascend-kg-config.json file is not recommended, as it may cause the MindCluster Ascend FaultDiag component to malfunction.

   ascend-fd entity --check custom-ascend-kg-config.json

   The following is an example of the output, indicating that the file verification is successful.

   Custom entity verification passed.