yolov5多batch模型推理相比单batch没有缩短
收藏回复举报
yolov5多batch模型推理相比单batch没有缩短
t('forum.solved') 已解决
发表于2023-03-13 10:51:01
0 查看

一、问题现象(附报错日志上下文):
yolov5 的模型转换为多batch,相比单batch平均推理速度并无提升

二、硬件及软件版本:

芯片类型 Ascend 310(npu-smi 22.0.4      Version: 22.0.4)
-- CANN 版本 (6.0.0):
使用PYACL
--Python 版本 (python=3.9):
--操作系统版本 (e.g., Ubuntu 18.04):

三、测试步骤:
从原始的yolov5n.pt出发,利用yolov5官方的export文件,分别使用
python export.py --weights yolov5n.pt --opset 12 --simplify --include onnx --batch-size 1
python export.py --weights yolov5n.pt --opset 12 --simplify --include onnx --batch-size 16 (--dynamic,--opset11均有尝试,全部不行)
进行onnx转换,得到yolov5_batch1.onnx,以及yolov5_batch16.onnx,然后再使用ATC转换命令,
atc --input_shape="images:1,3,640,640" --input_format=NCHW --output="ba1" --soc_version=Ascend310 --framework=5 --model="yolov5_batch1.onnx" --output_type=FP32

atc --input_shape="images:1,3,640,640" --input_format=NCHW --output="ba16" --soc_version=Ascend310 --framework=5 --model="yolov5_batch16.onnx" --output_type=FP32

得到ba1.om和ba16.om,这两个batch不同的模型,按理说,对于16个数据(图像),ba16.om的总推理时间应该要比ba1.om的总推理时间小不少,但是,我根据转换的模型发现ba16.om的总时间恰好约为ba1.om的16倍,即两个模型平均的推理时间相同。这与经验完全不相符,怀疑模型转换中算子不支持出现了问题?

四、日志信息:
推理时间
ba1.om
(base) root@ecs-b523:~/test# ./benchmark.aarch64 -device_id=0 -om_path=/root/detect_310/ascend/ba1.om -round=30 -batch_size=1
[INFO][Inference] PureInfer Init SUCCESS
[INFO] Dataset number: 0 finished cost 8.029ms
[INFO] Dataset number: 1 finished cost 7.963ms
[INFO] Dataset number: 2 finished cost 7.945ms
[INFO] Dataset number: 3 finished cost 7.919ms
[INFO] Dataset number: 4 finished cost 7.943ms
[INFO] Dataset number: 5 finished cost 7.941ms
[INFO] Dataset number: 6 finished cost 7.988ms
[INFO] Dataset number: 7 finished cost 7.926ms
[INFO] Dataset number: 8 finished cost 7.913ms
[INFO] Dataset number: 9 finished cost 7.917ms
[INFO] Dataset number: 10 finished cost 7.901ms
[INFO] Dataset number: 11 finished cost 7.954ms
[INFO] Dataset number: 12 finished cost 7.956ms
[INFO] Dataset number: 13 finished cost 7.913ms
[INFO] Dataset number: 14 finished cost 7.979ms
[INFO] Dataset number: 15 finished cost 7.953ms
[INFO] Dataset number: 16 finished cost 7.911ms
[INFO] Dataset number: 17 finished cost 7.98ms
[INFO] Dataset number: 18 finished cost 7.937ms
[INFO] Dataset number: 19 finished cost 7.961ms
[INFO] Dataset number: 20 finished cost 7.93ms
[INFO] Dataset number: 21 finished cost 7.918ms
[INFO] Dataset number: 22 finished cost 7.922ms
[INFO] Dataset number: 23 finished cost 7.926ms
[INFO] Dataset number: 24 finished cost 7.949ms
[INFO] Dataset number: 25 finished cost 7.937ms
[INFO] Dataset number: 26 finished cost 7.895ms
[INFO] Dataset number: 27 finished cost 7.979ms
[INFO] Dataset number: 28 finished cost 7.932ms
[INFO] Dataset number: 29 finished cost 7.967ms
[INFO] PureInfer result saved in ./result/PureInfer_perf_of_ba1_in_device_0.txt

ba16.om
(base) root@ecs-b523:~/test# ./benchmark.aarch64 -device_id=0 -om_path=/root/detect_310/ascend/ba16.om -round=30 -batch_size=16
[INFO][Inference] PureInfer Init SUCCESS
[INFO] Dataset number: 0 finished cost 126.398ms
[INFO] Dataset number: 1 finished cost 126.188ms
[INFO] Dataset number: 2 finished cost 126.152ms
[INFO] Dataset number: 3 finished cost 126.237ms
[INFO] Dataset number: 4 finished cost 125.952ms
[INFO] Dataset number: 5 finished cost 126.159ms
[INFO] Dataset number: 6 finished cost 126.102ms
[INFO] Dataset number: 7 finished cost 126.221ms
[INFO] Dataset number: 8 finished cost 126.266ms
[INFO] Dataset number: 9 finished cost 126.218ms
[INFO] Dataset number: 10 finished cost 126.229ms
[INFO] Dataset number: 11 finished cost 126.215ms
[INFO] Dataset number: 12 finished cost 126.279ms
[INFO] Dataset number: 13 finished cost 126.153ms
[INFO] Dataset number: 14 finished cost 126.143ms
[INFO] Dataset number: 15 finished cost 126.308ms
[INFO] Dataset number: 16 finished cost 126.194ms
[INFO] Dataset number: 17 finished cost 126.081ms
[INFO] Dataset number: 18 finished cost 126.072ms
[INFO] Dataset number: 19 finished cost 126.229ms
[INFO] Dataset number: 20 finished cost 126.192ms
[INFO] Dataset number: 21 finished cost 126.223ms
[INFO] Dataset number: 22 finished cost 126.258ms
[INFO] Dataset number: 23 finished cost 126.257ms
[INFO] Dataset number: 24 finished cost 126.277ms
[INFO] Dataset number: 25 finished cost 126.179ms
[INFO] Dataset number: 26 finished cost 126.265ms
[INFO] Dataset number: 27 finished cost 126.223ms
[INFO] Dataset number: 28 finished cost 126.194ms
[INFO] Dataset number: 29 finished cost 126.257ms
[INFO] PureInfer result saved in ./result/PureInfer_perf_of_ba16_in_device_0.txt
-----------------PureInfer Performance Summary------------------
[INFO] ave_throughputRate: 126.778samples/s, ave_latency: 7.88775ms

我要发帖子