Troubleshooting Workflow

Dump is enabled by using the AscendCL API. For details, see Accuracy Debugging Tool Guide.

1. Set your model to high-precision mode, re-execute model inference, and obtain dump data. If the inference accuracy meets requirements, go to 2.b; otherwise, go to 3.

   The high-precision mode might affect the inference performance. To improve the accuracy while preserving the performance, perform 2.b–2.d.

   For details about how to set the high-precision mode, see Setting the Model to High-Precision Mode.

2. Use the Model Accuracy Analyzer to compare the dump data obtained in high-precision mode with that obtained in 1.

   For details about how to use the tool, see Accuracy Debugging Tool Guide.

3. Locate the operators that show different calculation results according to 2.b.

   Locate the operator with the lowest cosine similarity (for example, lower than 0.95) and go to 2.d to run inference and dump your model.

   Repeat this step until no operator shows different calculation results.

4. For the operator with the lowest cosine similarity, preserve its original precision, and re-execute model inference.

   For details about how to preserve the precision of selected operators, see Preserving the Precision of Selected Operators.

1. Disable the buffer optimization function and re-execute model inference. If the inference accuracy meets requirements, go to 3.b; otherwise, go to 4.

   Buffer optimization is enabled by default to improve computing efficiency and performance. However, it is possible that your model contains an operator that is not yet covered by the current implementation. If the inference accuracy meets requirements after the buffer optimization function is disabled, submit the error operator to technical support, who will add buffer optimization support to your operator as soon as possible. After obtaining the logs, click here to contact technical support.

   For details about how to disable the buffer optimization function, see Disabling Buffer Optimization.

2. Use the Model Accuracy Analyzer to compare the dump data obtained after buffer optimization is disabled with that obtained in 1.

   For details about how to use the tool, see Accuracy Debugging Tool Guide.

3. Locate the operators that show different calculation results according to 3.b.
4. Submit these operators to technical support for further analysis. After obtaining the logs, click here to contact technical support.

1. Enable buffer optimization, disable fusion patterns, and re-execute model inference. If the inference accuracy meets requirements, go to 4.b; otherwise, go to 5.

   Fusion is enabled by default to improve computing efficiency and performance. However, it is possible that a fusion operator on your network is not yet covered by the current implementation. If the inference accuracy meets requirements after the fusion function is disabled, submit the error operator to technical support, who will add implementation support to your operator as soon as possible. After obtaining the logs, click here to contact technical support.

   For details about how to disable the fusion function, see Disabling Fusion. Disabling certain fusion patterns may cause functionality problems. Therefore, the system disables partial instead of all fusion patterns.

2. Use the Model Accuracy Analyzer to compare the dump data obtained after fusion patterns are disabled with that obtained in 1.

   For details about how to use the tool, see Accuracy Debugging Tool Guide.

3. Locate the operators that show different calculation results according to 4.b.
4. Submit these operators to technical support for further analysis. After obtaining the logs, click here to contact technical support.

Enable the fusion function, check the dataset, AIPP, and postprocessing configuration. For details, see Checking Data Processing Configuration.

If any data processing configuration is improper, correct your configuration and re-execute model inference. Otherwise, go to 6.

1. Use the Model Accuracy Analyzer to compare the dump data of the model with that of the benchmark network.

   For details about how to use the tool, see Accuracy Debugging Tool Guide.

2. Locate the operators that show different calculation results according to 6.a.
3. Submit these operators to technical support for further analysis. After obtaining the logs, click here to contact technical support.