Gradient Splitting Tuning

Tuning Precautions

1. Ensure that the training script can be successfully executed on the Ascend AI Processor and the function and accuracy meet the expectation.
2. Before gradient splitting tuning, disable the profiling function and the static memory allocation mode (that is, set static_memory_policy to 0, for example, custom_op.parameter_map["static_memory_policy"].i = 0) to avoid affecting the tuning result. Gradient splitting tuning supports only the static graph mode. For details about how to disable the profiling function, see the Performance Tuning Tool User Guide .
3. Before performing gradient splitting tuning, ensure that the training script is a distributed script.
4. If the gradient splitting tuning has been performed on the current network and the repository information has been generated, you do not need to enable the tuning again.
5. AOE does not allow different users to use the same device for tuning at the same time.
6. If there is only one AOE process, ensure that the available disk space in the home directory of the user who performs tuning is greater than or equal to 20 GB. If there are multiple AOE processes, increase the disk space accordingly.
7. Gradient splitting tuning does not support model parallelism.

Environment Variable Configuration

Tuning Procedure

1. Ensure that the collective communication operators exist during the training when the script is executed on a single device.

   Check whether the script contains either of the following settings:

   • Manually inserted collective communication operator, for example:

     from npu_bridge.hccl import hccl_ops
     from npu_bridge.npu_init import *
     if get_npu_rank_size() > 0:
        result = hccl_ops.allreduce(tensor, "sum")

   • Collective communication operator inserted by the distributed optimizer, for example:

     from npu_bridge.estimator.npu.npu_optimizer import NPUOtimizer
     from npu_bridge.npu_init import *
     # Define the optimizer.
     optimizer = LAMBOptimizer(......)
     optimizer = NPUOptimizer(optimizer,is_distributed=true)

2. Run the distributed training script on a single device.

   The following uses ResNet50_HC on device 0 as an example:

   "for((RANK_ID_n=$RANK_ID_START;RANK_ID_n<$((RANK_SIZE+RANK_ID_START));RANK_ID_n++))"

   Change it to:

   for((RANK_ID_n=$RANK_ID_START;RANK_ID_n<$((1+RANK_ID_START));RANK_ID_n++))

3. Run the training script to generate a custom repository.
   Key log information about tuning during training is as follows:

   # Enable TFAdapter tuning.
   in tune mode, training graph handled by tools
   # Start the tool for tuning.
   Aoe tuning graph.

   The generated custom repository is stored in the path specified by the TUNE_BANK_PATH environment variable. If this environment variable is not configured, the custom repository is stored in the ${HOME}/Ascend/latest/data/aoe/custom/graph/${soc_version} directory by default. For details about how to use the tuned custom repository, see Usage of Tuned Custom Repositories.

4. After the training is complete, delete the AOE_MODE environment variable (command: unset AOE_MODE) to disable the tuning mode. To use the custom repository, ensure that the TUNE_BANK_PATH environment variable is valid.
5. Restore the training script running on a single device to a distributed training script and perform distributed training. The network implements AllReduce fusion based on the gradient splitting policy in the repository.
6. Key log information about the gradient splitting points calculated by the repository during training is as follows:

   1	Use fusion library value

Restrictions

from hccl.split.api import set_split_strategy_by_idx
set_split_strategy_by_idx([118,159,160])