Tensor Decomposition

Tensor decomposition converts a convolution into a stack of two smaller ones by decomposing its convolution kernel to reduce the inference overhead. If the user model involves huge convolution workloads and most of the convolution kernels have shapes larger than (64, 64, 3, 3), tensor decomposition is recommended. In other cases, skip this step and proceed to quantization.

Currently, tensor decomposition is supported under the following conditions:

group = 1, dilation = (1,1), stride < 3
kernel_h > 2, kernel_w > 2

Only when the original Caffe model has a Convolution layer and the layer meets the preceding conditions, the Convolution layer can be decomposed into two smaller Convolution layers. Then, you can use AMCT to convert the original Caffe model into a quantizable model deployable on the AI processor for better inference performance.

This step is optional.

Restrictions

For Convolution kernels with large shapes, the decomposition process might be time-consuming or terminated abnormally. To prevent this problem, refer to the following before starting decomposition:

Reference performance specifications of the decomposition tool:
- CPU: Intel(R) Xeon(R) CPU E5-2699 v4@2.20 GHz
- Memory: 512 GB
Time taken to decompose a single convolutional layer:
- About 25s for shape (512, 512, 5, 5).
- About 16s for shape (1024, 1024, 3, 3).
- About 78s for shape (1024, 1024, 5, 5).
- About 63s for shape (2048, 2048, 3, 3).
- About 430s for shape (2048, 2048, 5, 5).
Memory consideration:
It takes about 32 GB memory to decompose a convolution kernel with shape (2048, 2048, 5, 5).

API Call Sequence

Figure 1 shows the API call sequence. For the decomposition example, see Sample List.

Figure 1 API call sequence for tensor decomposition

The workflow goes through the following steps:

Call the auto_decomposition API to perform tensor decomposition on the original Caffe model, generating a new model file and a new weight file.
Fine-tune the decomposed model. Optionally quantize the fine-tuning model (see PTQ or QAT for details).

Figure 2 shows the resnet50 model before and after decomposition.

Figure 2 Model before and after decomposition

Example

      
           # Related module import
from amct_caffe.tensor_decompose import auto_decomposition

# Original model file
model_file = 'src_path/xxx.prototxt'    
# Original weight file
weights_file = 'src_path/xxx.caffemodel'  
# Resultant model file
new_model_file = 'decomposed_path/xxx.prototxt'   
# Resultant weight file
new_weights_file = 'decomposed_path/xxx.caffemodel' 

# Tensor decomposition
auto_decomposition(model_file=model_file, weights_file=weights_file,
                   new_model_file=new_model_file, new_weights_file=new_weights_file)

Parent topic: AMCT (Caffe)