compare Accuracy Comparison

Prerequisites

• You have performed operations in Environment Setup.
• Taking the cell module comparison in different MindSpore versions as an example. You have dumped the cell modules of different framework versions as instructed in Accuracy Data Collection. MindSpore 2.6.0 and MindSpore 2.7.0 are used as examples.

Performing Comparison

1. Prepare data.

   Install MindSpore 2.6.0 and MindSpore 2.7.0 in the same Ascend NPU environment and perform the dump operation to obtain two copies of accuracy data. Pay attention to the directory names specified by dump_path. dump_data_2.6.0 and dump_data_2.7.0 are used as examples.

2. Create a configuration file for comparison.
   For example, create a compare.json configuration file in the directory where the training script is located and copy the following content to the file:

   1 2 3 4 5 6

   { "npu_path": "./dump_data_2.7.0/step0/rank/dump.json", "bench_path": "./dump_data_2.6.0/step0/rank/dump.json", "stack_path": "./dump_data_2.7.0/step0/rank/stack.json", "is_print_compare_log": true }

   The paths specified by npu_path and bench_path must be in the same environment.

3. Perform the comparison.
   The command is as follows:

   msprobe -f mindspore compare -i ./compare.json -o ./compare_result/accuracy_compare -s

   If the following information is displayed, the comparison is successful:

   1 2 3 4 5 6 7

   ... Compare result is /xxx/compare_result/accuracy_compare/compare_result_{timestamp}.xlsx ... The advisor summary is saved in: /xxx/compare_result/accuracy_compare/advisor_{timestamp}.txt ************************************************************************************ * msprobe compare ends successfully. * ************************************************************************************

4. Analyze the comparison result file.

   The following files are generated in ./compare_result/accuracy_compare:

   • advisor_{timestamp}.txt: provides expert suggestions on APIs that may have accuracy issues.
   • compare_result_{timestamp}.xlsx: lists the details about all APIs for accuracy comparison and the comparison results. You can locate suspicious operators based on the color, Result, Accuracy Reached or Not, and Err_Message (error message). However, each indicator has its own evaluation criteria. Therefore, you need to determine the result based on the actual situation.
     Example:

     Figure 1 compare_result_1
     

     Figure 2 compare_result_2
     

   For details about the comparison result analysis, see Accuracy Comparison Result Analysis.