What Do I Do If "Check scale and offset record file record.txt failed" Is Displayed During Quantization?

Symptom

When the save_model API is called to save the quantization model during quantization, the activation quantization parameters scale_d and offset_d computed in the calibration phase need to be read. If the parameters cannot be found in the corresponding record file, the quantization model fails to be saved. Therefore, the preceding AMCT error is thrown and the process is terminated as follows:

2021-05-11 09:29:18,138 - INFO - [AMCT]:[Optimizer]: Do <class 'amct_caffe.optimizer.check_record_scale_offset.CheckRecordScaleOffsetPass'>
2021-05-11 09:29:18,138 - ERROR - [AMCT]:[AMCT]: Cannot find scale_d,offset_d,channels,height,width of layer:Convolution1
2021-05-11 09:29:18,138 - ERROR - [AMCT]:[AMCT]: Check scale and offset record file xx/data/AMCT_CAFFE_GPU_GPU_FAQ/AMCT_CAFFE_GPU_GPU_FAQ_record.txt failed
2021-05-11 09:29:18,138 - ERROR - [AMCT]:[AMCT]: There may be something wrong while doing calibration
2021-05-11 09:29:18,138 - ERROR - [AMCT]:[AMCT]: Please check caffe log
Traceback (most recent call last):
  File "test_caffe_faq.py", line 32, in <module>
    main()
  File "test_caffe_faq.py", line 28, in main
    gen_amct_model(case,json_flag = True) 
  File "../caffe_lib/calibration_test/code/gen_amct_model.py", line 102, in gen_amct_model
    save_model(graph, mode, AMCT_PATH+case_name)
  File "xx/amct/lib/python3.7/site-packages/amct_caffe/common/utils/check_params.py", line 43, in wrapper
    return func(*args, **kwargs)
  File "xx/amct/lib/python3.7/site-packages/amct_caffe/quantize_tool.py", line 190, in save_model
    optimizer.do_optimizer(graph)
  File "xx/amct/lib/python3.7/site-packages/amct_caffe/optimizer/graph_optimizer.py", line 60, in do_optimizer
    graph_pass.run(graph)
  File "xx/amct/lib/python3.7/site-packages/amct_caffe/optimizer/check_record_scale_offset.py", line 103, in run
    raise RuntimeError('Check file {} failed.'.format(record_file))
RuntimeError: Check file xx/data/AMCT15_CAFFE_GPU_GPU_FAQ/AMCT15_CAFFE_GPU_GPU_FAQ_record.txt failed.

Possible Cause

The scale_d and offset_d parameters are saved during calibration (when the Caffe framework is called to perform forward propagation on the calibration model). AMCT inserts the IFMR layer into the calibration model, and the IFMR layer needs to accumulate enough user-specified batch_num data and then perform quantization to obtain scale_d and offset_d. The possible causes of the RuntimeError: Check file record.txt failed error are as follows:

An error occurs when running inference on the Caffe model. The possible causes are as follows: The built Caffe is faulty, the calibration model is faulty, or the corresponding dataset cannot be found. You can inspect the error message thrown by the Caffe framework.
The data amount of the calibration dataset provided by the user does not meet that required by batch_num. For example, if the user provides only one batch of data as the calibration dataset and batch_num is set to 2, the IFMR layer does not store sufficient data during calibration, the quantization fails. As a result, scale_d and offset_d cannot be computed, and the preceding error is reported. You can view the process information printed during the quantization of the IFMR to locate the fault. The amount of saved data is displayed in the IFMR layer.

When the specified amount of data is accumulated, the quantization operation is triggered.

The quantization of the current layer is complete only when "Do layer:"conv1" activation calibration success!" is displayed.

Solution

Rectify the fault based on the error reported by the Caffe framework.
Enlarge the calibration dataset or decrease the value of batch_num for the quantization algorithm until the data size of the calibration dataset is greater than or equal to the value of batch_num. Note that decreasing the value of batch_num may reduce the accuracy of the quantized model.

Parent topic: FAQs