Pre-Training Configuration Check
Based on the samples in this document, you need to add the tool APIs to the training script for configuration check.
The following example compares accuracy between two different MindSpore versions in the same environment. The only difference between the two scenarios is the version number. You can skip this step.
Prerequisites
- You have performed operations in Environment Setup.
- You have performed operations in Model Development and Migration. Ensure that the training job can be properly executed in all sample environments.
Environment Setup
Run the following command to install msprobe in the Ascend NPU environment:
pip install mindstudio-probe
Performing a Check
The procedure is as follows:
- Obtain the .zip packages of the two environments (including the environment configurations that affect the training accuracy: environment variables, third-party library versions, training hyperparameters, weights, datasets, and random operations).
Perform the following operations on MindSpore 2.6.0 and MindSpore 2.7.0, respectively. Name the two .zip packages differently.
You can copy the complete code from Code Sample for MindSpore Pre-Training Configuration Check and execute it directly. The following examples only show where to add the tool API in the script.
- Insert the following code at the beginning of the first Python script executed in the training process:
1 2
1 from msprobe.core.config_check import ConfigChecker 2 ConfigChecker.apply_patches(fmk)
fmk is a string parameter indicating the training framework, which can be set to pytorch or mindspore. In this example, mindspore is used.
- Insert the following code after the model is initialized:
1 2
49 from msprobe.core.config_check import ConfigChecker 50 ConfigChecker(model, output_zip_path, fmk)
- model indicates the initialized model. By default, the weight and dataset are not collected.
- output_zip_path indicates the path of the output .zip package. The value is of the string type. You need to specify the name of the .zip package. The default value is ./config_check_pack.zip.
- fmk indicates the training framework. The value can be set to pytorch or mindspore. In this example, mindspore is used.
- Run the training script.
python mindspore_main.py
After the collection is complete, a .zip package is generated, which contains the configurations that affect the accuracy. The data is stored by rank and step (micro_step).
- Insert the following code at the beginning of the first Python script executed in the training process:
- Upload the two .zip packages to the same environment and run the following command to compare them:
msprobe -f mindspore config_check -c bench_zip_path cmp_zip_path -o output_path
bench_zip_path indicates the name of the .zip package collected on the benchmark side, and cmp_zip_path indicates the name of the .zip package collected on the comparison side.
The default value of output_path is ./config_check_result.
After the preceding command is executed, the following data is generated in output_path:
- bench indicates the data packaged in bench_zip_path.
- cmp indicates the data packaged in cmp_zip_path.
- result.xlsx indicates the comparison results. There are multiple sheets. The summary sheet shows the overall results, and other sheets show the details of specific check items. The step is micro_step.
- Check the results.If the following five items are the same in both environments, the check passes. If they differ, adjust the environment accordingly.
- Environment variable
- Third-party library version
- Dataset
- Weight
- Training hyperparameter
See the following example:
1 2 3 4 5 6
filename ass_check env TRUE pip FALSE dataset TRUE weights TRUE random TRUE