Auto Channel Pruning Search Algorithm

By default, the greedy search algorithm is used for auto channel pruning search. Under the constraints of the given model calculation amount, the system automatically selects the removable channels at each layer (channels with lower order). This reduces the impact of channel pruning on the precision loss of the model after fine-tuning.

The problem is a combinatorial optimization problem, and is modeled as a knapsack problem. The value vi and the weight Fi of each item are known, and the item with the largest total value can be loaded under the knapsack capacity C is calculated, which may be expressed by using the following formula:

Maximization:

Constrained by:

vi is equivalent to sparsity sensitivity; Fi is bit complexity; b is a channel sparsity solution, that is, whether the channel is reserved; and C is a compression rate set by a user.

For a channel sparsity task, the weight Fi is the calculation amount of the ith channel, and vi is the loss function change of the network w after the ith channel is tailored:

Loss (w - wi) may be approximately estimated by using a first-order or second-order Taylor expansion.

The value density (sparsity sensitivity/bit complexity) of each channel at each layer is calculated based on the target sparsity rate specified by the user. After global sorting, a batch of channels with the lowest value density are selected for tailoring. The preceding combinatorial optimization method is used to solve the problem and complete the search task.