compare Accuracy Comparison

Prerequisites

• You have performed operations in Environment Setup.
• You have performed operations in Accuracy Data Collection and obtained the accuracy data of the GPU and Ascend NPU environments.

Performing Comparison

1. Prepare data.

   After dumping data in the GPU and Ascend NPU environments, copy the accuracy data dumped from the GPU environment to the Ascend NPU environment. Pay attention to the directory names specified by dump_path. dump_data_npu and dump_data_gpu are used as examples.

2. Create a configuration file for comparison.
   For example, create a compare.json configuration file in the directory where the training script is located and copy the following content to the file:

   1 2 3 4 5 6

   { "npu_path": "./dump_data_npu/step0/rank/dump.json", "bench_path": "./dump_data_gpu/step0/rank/dump.json", "stack_path": "./dump_data_npu/step0/rank/stack.json", "is_print_compare_log": true }

   The paths specified by npu_path and bench_path must be in the same environment.

3. Perform the comparison.
   The command is as follows:

   msprobe -f pytorch compare -i ./compare.json -o ./compare_result/accuracy_compare -s

   If the following information is displayed, the comparison is successful:

   1 2 3 4 5 6 7

   ... Compare result is /xxx/compare_result/accuracy_compare/compare_result_{timestamp}.xlsx ... The advisor summary is saved in: /xxx/compare_result/accuracy_compare/advisor_{timestamp}.txt ************************************************************************************ * msprobe compare ends successfully. * ************************************************************************************

4. Analyze the comparison result file.

   The following files are generated in ./compare_result/accuracy_compare:

   • advisor_{timestamp}.txt: provides expert suggestions on APIs that may have accuracy issues.
   • compare_result_{timestamp}.xlsx: lists the details about all APIs for accuracy comparison and the comparison results. You can locate suspicious operators based on the color, Result, Accuracy Reached or Not, and Err_Message (error message). However, each indicator has its own evaluation criteria. Therefore, you need to determine the result based on the actual situation.
     Example:

     Figure 1 compare_result_1
     

     Figure 2 compare_result_2
     

   For details about the comparison result analysis, see Accuracy Comparison Result Analysis.