[EVENT] PROFILING(5140,python):2024-05-10-18:13:24.462.775 [msprof_callback_impl.cpp:199] >>> (tid:5140) Started to register profiling ctrl callback.
[EVENT] PROFILING(5140,python):2024-05-10-18:13:24.906.493 [msprof_callback_impl.cpp:78] >>> (tid:5140) MsprofCtrlCallback called, type: 255
[EVENT] PROFILING(5140,python):2024-05-10-18:13:24.906.552 [prof_acl_mgr.cpp:1190] >>> (tid:5140) Init profiling for dynamic profiling
[EVENT] TDT(5140,python):2024-05-10-18:13:24.906.651 [log.cpp:43][ProcessModeManager] enter into open process deviceId[0] rankSize[0],[process_mode_manager.cpp:62:Open]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.906.891 [log.cpp:43][TsdClient] deviceId[0] begin to init hdc client,[process_mode_manager.cpp:307:InitTsdClient]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.907.089 [log.cpp:43]VersionVerify: send client version to server,[version_verify.cpp:34:SetVersionInfo]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.907.099 [log.cpp:43]send feature_info:{msg_type:35, features:{check before send aicpu package,}},[version_verify.cpp:51:SetVersionInfo]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.907.105 [log.cpp:43]send feature_info:{msg_type:37, features:{check before send open qs message,}},[version_verify.cpp:51:SetVersionInfo]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.907.612 [log.cpp:43]VersionVerify: Check client version info, server[1230], client[1230],[version_verify.cpp:67:PeerVersionCheck]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.907.627 [log.cpp:43]VersionVerify: pass client version info success,[version_verify.cpp:88:ParseVersionInfo]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.907.632 [log.cpp:43]Service[2] create hdc success,[hdc_client.cpp:264:CheckHdcConnection]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.907.640 [log.cpp:43]VersionVerify: new type[35], supported,[version_verify.cpp:121:SpecialFeatureCheck]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.907.651 [log.cpp:43][TsdClient][deviceId=0] [sessionId=1]wait package info respond,[process_mode_manager.cpp:606:GetDeviceCheckCode]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.907.918 [log.cpp:43][TsdClient] deviceId[0] begin to init hdc client,[process_mode_manager.cpp:307:InitTsdClient]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.911.766 [log.cpp:43]VersionVerify: send client version to server,[version_verify.cpp:34:SetVersionInfo]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.911.778 [log.cpp:43]send feature_info:{msg_type:35, features:{check before send aicpu package,}},[version_verify.cpp:51:SetVersionInfo]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.911.784 [log.cpp:43]send feature_info:{msg_type:37, features:{check before send open qs message,}},[version_verify.cpp:51:SetVersionInfo]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.912.095 [log.cpp:43]VersionVerify: Check client version info, server[1230], client[1230],[version_verify.cpp:67:PeerVersionCheck]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.912.104 [log.cpp:43]VersionVerify: pass client version info success,[version_verify.cpp:88:ParseVersionInfo]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.912.110 [log.cpp:43]Service[2] create hdc success,[hdc_client.cpp:264:CheckHdcConnection]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.912.121 [log.cpp:43][TsdClient] tsd get process sign successfully, procpid[5140] signSize[48],[process_mode_manager.cpp:353:SendOpenMsg]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.912.128 [log.cpp:43]VersionVerify: previous type[6], supported,[version_verify.cpp:113:SpecialFeatureCheck]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:24.912.140 [log.cpp:43][ProcessModeManager] deviceId[0] sessionId[1] rankSize[0], wait sub process start respond,[process_mode_manager.cpp:84:Open]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:25.044.860 [log.cpp:43][TsdClient][deviceId=0] [sessionId=1] start hccp and computer process success,[process_mode_manager.cpp:96:Open]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:25.044.876 [log.cpp:43][TsdClient] it is unneccessary of current mode[0] chiptype[2] to grant queue auth to aicpusd,[process_mode_manager.cpp:101:Open]5140
[EVENT] TDT(5140,python):2024-05-10-18:13:25.055.997 [log.cpp:43][TsdClient] set profiling callback success.,[client_manager.cpp:158:SetProfilingCallback]5140
[EVENT] GE(5140,python):2024-05-10-18:13:25.177.375 [davinci_model.cc:382]5140 InitFeatureMapAndP2PMem:[IMAS]InitFeatureMapAndP2PMem graph_0 MallocMemory type[F] memaddr[108060000000] mem_size[506954752]
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.553.994 [engine.cc:1101]5147 ReportExceptProc:Task exception! device_id=0, stream_id=13, task_id=1, type=13, retCode=0x91, [the model stream execute failed].
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.097 [device_error_proc.cc:495]5147 PrintCoreErrorInfo:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.105 [device_error_proc.cc:495]5147 PrintCoreErrorInfo:The error from device(0), serial number is 14, there is an aicore error, core id is 0, error code = 0x10, dump info: pc start: 0x1080802a2000, current: 0x1080802a2218, vec error info: 0x62ef6f6, mte error info: 0xef, ifu error info: 0x130cb0801380, ccu error info: 0x2777077600154181, cube error info: 0x1f, biu error info: 0, aic error mask: 0x65000200d000288, para base: 0x10808001e378, errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses.
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.161 [device_error_proc.cc:495]5147 PrintCoreErrorInfo:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.165 [device_error_proc.cc:495]5147 PrintCoreErrorInfo:The error from device(0), serial number is 14, there is an aicore error, core id is 1, error code = 0x10, dump info: pc start: 0x1080802a2000, current: 0x1080802a21cc, vec error info: 0x2072623, mte error info: 0x58, ifu error info: 0x39fb9b819b80, ccu error info: 0x344a33620037266c, cube error info: 0x1f, biu error info: 0, aic error mask: 0x65000200d000288, para base: 0x10808001e378, errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses.
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.277 [task.cc:1035]5147 PrintErrorInfo:Aicore kernel execute failed, device_id=0, stream_id=1, report_stream_id=13, task_id=229, flip_num=0, fault kernel_name=1735795989364282369-1_0_1_PartitionedCall_/Gather_12_gatherv2_156, func_name=te_gatherv2_b967f8d9285149b4e2fcc8d281cd4bd59b8957d1bea598991cf77b4ea6d89170_1__kernel0, program id=216, hash=13058336217577453705.
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.290 [task.cc:3286]5147 ReportErrorInfo:model execute error, retCode=0x91, [the model stream execute failed].
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.295 [task.cc:3258]5147 PrintErrorInfo:model execute task failed, device_id=0, model stream_id=13, model task_id=1, flip_num=0, model_id=2, first_task_id=65535
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.341 [stream.cc:918]5140 GetError:[EXEC][DEFAULT]Stream Synchronize failed, stream_id=13, retCode=0x91, [the model stream execute failed].
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.373 [stream.cc:921]5140 GetError:[EXEC][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.378 [stream.cc:921]5140 GetError:[EXEC][DEFAULT]Aicore kernel execute failed, device_id=0, stream_id=1, report_stream_id=13, task_id=229, flip_num=0, fault kernel_name=1735795989364282369-1_0_1_PartitionedCall_/Gather_12_gatherv2_156, func_name=te_gatherv2_b967f8d9285149b4e2fcc8d281cd4bd59b8957d1bea598991cf77b4ea6d89170_1__kernel0, program id=216, hash=13058336217577453705
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.413 [model.cc:581]5140 SynchronizeExecute:[EXEC][DEFAULT]Fail to synchronize forbbiden stream_id=13, retCode=0x7150050!
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.420 [model.cc:605]5140 GetStreamToSyncExecute:[EXEC][DEFAULT]report error module_type=0, module_name=EE9999
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.425 [model.cc:605]5140 GetStreamToSyncExecute:[EXEC][DEFAULT]Model synchronize execute failed, model_id=2!
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.443 [logger.cc:847]5140 ModelExecute:[EXEC][DEFAULT]Execute model failed.
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.467 [api_c.cc:2053]5140 rtModelExecute:[EXEC][DEFAULT]ErrCode=507011, desc=[the model stream execute failed], InnerCode=0x7150050
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.472 [error_message_manage.cc:49]5140 FuncErrorReason:[EXEC][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(5140,python):2024-05-10-18:13:25.555.479 [error_message_manage.cc:49]5140 FuncErrorReason:[EXEC][DEFAULT]rtModelExecute execute failed, reason=[the model stream execute failed]
[ERROR] GE(5140,python):2024-05-10-18:13:25.555.494 [davinci_model.cc:4551]5140 NnExecute: ErrorNo: 1343225859(Failed to call runtime API!) [EXEC][DEFAULT]Call rt api failed, ret: 0x7BC83
[ERROR] GE(5140,python):2024-05-10-18:13:25.555.504 [graph_loader.cc:235]5140 ExecuteModel: ErrorNo: 507011() [EXEC][DEFAULT][Execute][Model] failed, model_id:1.
[ERROR] ASCENDCL(5140,python):2024-05-10-18:13:25.555.513 [model.cpp:750]5140 ModelExecute: [EXEC][DEFAULT][Exec][Model]Execute model failed, ge result[507011], modelId[1]
[ERROR] ASCENDCL(5140,python):2024-05-10-18:13:25.555.528 [model.cpp:1592]5140 aclmdlExecute: [EXEC][DEFAULT][Exec][Model]modelId[1] execute failed, result[507011]
Exception ignored in: <function RTMDet_ascend.__del__ at 0x7fd65b255f70>
Traceback (most recent call last):
  File "rtmdet.py", line 197, in __del__
    acl.rt.free(self.input_buffer)
AttributeError: 'RTMDet_ascend' object has no attribute 'input_buffer'
4.05 2592 1944
execute failed, errorCode is 507011
[EVENT] IDEDH(5140,python):2024-05-10-18:13:25.608.319 [adx_server_manager.cpp:27][tid:5140]>>> start to deconstruct adx server manager

import os

import acl
import numpy as np
from PIL import Image
import cv2
import mmcv


NPY_FLOAT32 = 11
ACL_MEMCPY_HOST_TO_HOST = 0
ACL_MEMCPY_HOST_TO_DEVICE = 1
ACL_MEMCPY_DEVICE_TO_HOST = 2
ACL_MEMCPY_DEVICE_TO_DEVICE = 3
ACL_MEM_MALLOC_HUGE_FIRST = 0
ACL_DEVICE, ACL_HOST = 0, 1
ACL_SUCCESS = 0

def aspect_resize(frame, target_w, target_h):
    raw_w, raw_h = frame.shape[1], frame.shape[0]
    ratio = max(raw_w / target_w, raw_h / target_h)
    resized_w, resized_h = int(raw_w / ratio), int(raw_h / ratio)

    frame = cv2.resize(frame, (resized_w, resized_h))
    resized_frame = np.full((target_h, target_w, 3), 112, dtype=float)
    print(ratio, raw_w, raw_h)
    resized_frame[:resized_h, :resized_w, :] = frame
    return resized_frame, ratio
    
class RTMDet_ascend:

    def __init__(self, model_path='./onnx_om.om', device_id=0):
        # 初始化函数
        self.device_id = device_id
        self.runMode_ = acl.rt.get_run_mode()
        
        # step1: 初始化
        ret = acl.init()
        if ret != ACL_SUCCESS:
            print('acl init failed, errorCode is', ret)
            
        # 指定运算的Device
        ret = acl.rt.set_device(self.device_id)
        if ret != ACL_SUCCESS:
            print('set device failed, errorCode is', ret)

        self.context, ret = acl.rt.create_context(self.device_id)
        if ret != ACL_SUCCESS:
            print('create context failed, errorCode is', ret)


        self.stream, ret = acl.rt.create_stream()
        if ret != ACL_SUCCESS:
            print('create stream failed, errorCode is', ret)
        
        # step2: 加载模型，本示例为ResNet-50模型
        # 加载离线模型文件，返回标识模型的ID
        self.model_id, ret = acl.mdl.load_from_file(model_path)
        if ret != ACL_SUCCESS:
            print('load model failed, errorCode is', ret)
            
        # 创建空白模型描述信息，获取模型描述信息的指针地址
        self.model_desc = acl.mdl.create_desc()
        # 通过模型的ID，将模型的描述信息填充到model_desc
        ret = acl.mdl.get_desc(self.model_desc, self.model_id)
        if ret != ACL_SUCCESS:
            print('get desc failed, errorCode is', ret)

        # step3：创建输入输出数据集
        # 创建输入数据集
        self.input_dataset, self.input_data = self.prepare_dataset('input')
        # 创建输出数据集
        self.output_dataset, self.output_data = self.prepare_dataset('output')
        
        # # create data set of input
        # self.input_dataset = acl.mdl.create_dataset()
        # input_index = 0
        # self.input_buffer_size = acl.mdl.get_input_size_by_index(self.model_desc, input_index)
        # self.input_buffer, ret = acl.rt.malloc(self.input_buffer_size, ACL_MEM_MALLOC_HUGE_FIRST)
        # input_data = acl.create_data_buffer(self.input_buffer, self.input_buffer_size)
        # self.input_dataset, ret = acl.mdl.add_dataset_buffer(self.input_dataset, input_data)
        # if ret != ACL_SUCCESS:
        #     print('acl.mdl.add_dataset_buffer failed, errorCode is', ret)

        # # create data set of output
        # self.output_dataset = acl.mdl.create_dataset()
        # output_index = 0
        # self.output_buffer_size = acl.mdl.get_output_size_by_index(self.model_desc, output_index)
        # self.output_buffer, ret = acl.rt.malloc(self.output_buffer_size, ACL_MEM_MALLOC_HUGE_FIRST)
        # output_data = acl.create_data_buffer(self.output_buffer, self.output_buffer_size)
        # self.output_dataset, ret = acl.mdl.add_dataset_buffer(self.output_dataset, output_data)
        # if ret != ACL_SUCCESS:
        #     print('acl.mdl.add_dataset_buffer failed, errorCode is', ret)
    
    def prepare_dataset(self, io_type):
        # 准备数据集
        if io_type == "input":
            # 获得模型输入的个数
            io_num = acl.mdl.get_num_inputs(self.model_desc)
            acl_mdl_get_size_by_index = acl.mdl.get_input_size_by_index
        else:
            # 获得模型输出的个数
            io_num = acl.mdl.get_num_outputs(self.model_desc)
            acl_mdl_get_size_by_index = acl.mdl.get_output_size_by_index
        # 创建aclmdlDataset类型的数据，描述模型推理的输入。
        dataset = acl.mdl.create_dataset()
        datas = []
        for i in range(io_num):
            # 获取所需的buffer内存大小
            buffer_size = acl_mdl_get_size_by_index(self.model_desc, i)
            # 申请buffer内存
            buffer, ret = acl.rt.malloc(buffer_size, ACL_MEM_MALLOC_HUGE_FIRST)
            # 从内存创建buffer数据
            data_buffer = acl.create_data_buffer(buffer, buffer_size)
            # 将buffer数据添加到数据集
            _, ret = acl.mdl.add_dataset_buffer(dataset, data_buffer)
            if ret != ACL_SUCCESS:
                print('acl.mdl.add_dataset_buffer failed, errorCode is', ret)
            datas.append({"buffer": buffer, "data": data_buffer, "size": buffer_size})
        return dataset, datas
    
    def forward(self, cv2_frame):
        #TODO aspect resize
        cv2_frame, ratio = aspect_resize(cv2_frame, 640, 640)
        cv2_frame -= np.array([103.53, 116.28, 123.675])
        cv2_frame /= np.array([57.375, 57.12, 58.395])
        cv2_frame = cv2_frame.transpose((2, 0, 1))
        cv2_frame = np.reshape(cv2_frame, (1, 3, 640, 640))

        bytes_data = np.frombuffer(cv2_frame.tobytes())
            
        # pass image data to input data buffer
        if self.runMode_ == ACL_DEVICE:
            kind = ACL_MEMCPY_DEVICE_TO_DEVICE
        else:
            kind = ACL_MEMCPY_HOST_TO_DEVICE
        if "bytes_to_ptr" in dir(acl.util):
            bytes_data = bytes_data.tobytes()
            ptr = acl.util.bytes_to_ptr(bytes_data)
        else:
            ptr = acl.util.numpy_to_ptr(bytes_data)
        # ret = acl.rt.memcpy(self.input_buffer,
        #                     self.input_buffer_size,
        #                     ptr,
        #                     self.input_buffer_size,
        #                     kind)
        ret = acl.rt.memcpy(self.input_data[0]["buffer"],   # 目标地址 device
                             self.input_data[0]["size"],     # 目标地址大小
                             ptr,                      # 源地址 host
                             self.input_data[0]["size"],                # 源地址大小
                             kind)      # 模式:从host到device
        if ret != ACL_SUCCESS:
            print('memcpy failed, errorCode is', ret)

        # inference
        ret = acl.mdl.execute(self.model_id,
                              self.input_dataset,
                              self.output_dataset)
        if ret != ACL_SUCCESS:
            print('execute failed, errorCode is', ret)
            
        # get result from output data set
        output_index = 0
        output_data_buffer = acl.mdl.get_dataset_buffer(self.output_dataset, output_index)
        output_data_buffer_addr = acl.get_data_buffer_addr(output_data_buffer)
        output_data_size = acl.get_data_buffer_size(output_data_buffer)
        ptr, ret = acl.rt.malloc_host(output_data_size)

        # copy device output data to host
        if self.runMode_ == ACL_DEVICE:
            kind = ACL_MEMCPY_DEVICE_TO_HOST
        else:
            kind = ACL_MEMCPY_HOST_TO_HOST
        ret = acl.rt.memcpy(ptr,
                            output_data_size,
                            output_data_buffer_addr,
                            output_data_size,
                            ACL_MEMCPY_HOST_TO_DEVICE)
        if ret != ACL_SUCCESS:
            print('memcpy failed, errorCode is', ret)

        index = 0
        dims, ret = acl.mdl.get_cur_output_dims(self.model_desc, index)

        if ret != ACL_SUCCESS:
            print('get output dims failed, errorCode is', ret)
        out_dim = dims['dims']

        if "ptr_to_bytes" in dir(acl.util):
            bytes_data = acl.util.ptr_to_bytes(ptr, output_data_size)
            data = np.frombuffer(bytes_data, dtype=np.float32).reshape(out_dim)
        else:
            data = acl.util.ptr_to_numpy(ptr, out_dim, NPY_FLOAT32)
        
    def __del__(self):
        # release resource includes acl resource, data set and unload model
        # acl.rt.free(self.input_buffer)
        acl.mdl.destroy_dataset(self.input_dataset)

        # acl.rt.free(self.output_buffer)
        acl.mdl.destroy_dataset(self.output_dataset)
        ret = acl.mdl.unload(self.model_id)
        if ret != ACL_SUCCESS:
            print('unload model failed, errorCode is', ret)

        if self.model_desc:
            acl.mdl.destroy_desc(self.model_desc)
            self.model_desc = None

        if self.stream:
            ret = acl.rt.destroy_stream(self.stream)
            if ret != ACL_SUCCESS:
                print('destroy stream failed, errorCode is', ret)
            self.stream = None

        if self.context:
            ret = acl.rt.destroy_context(self.context)
            if ret != ACL_SUCCESS:
                print('destroy context failed, errorCode is', ret)
            self.context = None

        ret = acl.rt.reset_device(self.device_id)
        if ret != ACL_SUCCESS:
            print('reset device failed, errorCode is', ret)
            
        ret = acl.finalize()
        if ret != ACL_SUCCESS:
            print('finalize failed, errorCode is', ret)
        
if __name__=='__main__':
    model = RTMDet_ascend()
    f = cv2.imread('frame87.jpg')
    try:
        model.forward(f)
    finally:
        del model