Auto Channel Pruning Search Algorithm

By default, the greedy search algorithm is used for auto channel pruning search. Under the constraints of the given model calculation amount, the system automatically selects the removable channels at each layer (channels with lower order). This reduces the impact of channel pruning on the precision loss of the model after fine-tuning.

This is a combinatorial optimization problem, modeled as a knapsack problem. As the value v_i and weight F_i of all items are known, calculate the item with the largest total value under the knapsack capacity C. The formula is as follows:

Maximization:

Constrained by:

where v_i is equivalent to sparsity sensitivity; F_i is bit complexity; b is a channel pruning scheme, that is, whether the channel is reserved; and C is a compression ratio set by a user.

For a channel pruning task, the weight F_i is the calculation amount of the ith channel, and v_i is the change of the loss function of the network w after the ith channel is tailored:

loss(w - w_i) can be approximately estimated by using a first- or second-order Taylor expansion.

The value density (sparsity sensitivity/bit complexity) of each channel at each layer is calculated based on the target sparsity ratio specified by the user. After global sorting, a batch of channels with the lowest value density are selected for tailoring. Therefore, the preceding combinatorial optimization method solves the problem and completes the search task.