2:4 Structured Sparsity Algorithm

AMCT uses the 2:4 structured sparsity algorithm L1SelectivePrune to determine the weights to be reserved by comparing the l1 values (absolute values) of the weights. Among every four consecutive weights, the two weights with the largest l1 values are reserved.

In the sparse configuration file, the L1SelectivePruner field is used to control the L1SelectivePrune algorithm. (The PyTorch framework is used as an example. For details about the parameters, see.)

• update_freq: Indicates the interval for updating 2:4 sparsity, for calculating which elements are reserved. In a retraining process, the weight changes with each training batch, and the corresponding l1 value sequence may change. For example, the first two elements in the four elements are reserved, and the first and third elements may be reserved after the update.

  If update_freq is 0, the elements to be reserved are calculated only in the first batch during training. If update_freq is 2, the elements to be reserved are calculated in every two batches during training. The rest may be deduced by analogy. The default value is 0.

• N_out_of_m_type: Currently, only M4N2 is supported. That is, two weights are reserved in every four consecutive weights.