Queue-based Model Inference

This section describes how to load a model based on queues, prepare model input data, and obtain model inference output data.

Queue-based model inference uses queues to decouple inference jobs from data production and consumption and implement asynchronous scheduling. For example, data preprocessing and model inference use asynchronous queues to implement asynchronous parallelism.

Principles

Call aclmdlLoadFromFileWithQ or aclmdlLoadFromMemWithQ to load the model in queue mode.
Call the acltdtEnqueueData API to transfer the input data of the model to the queue. The API performs inference based on the input data in the queue. You do not need to call the model execution API.
Call acltdtDequeueData to wait until the model inference is complete, and then obtain the result data from the output buffer.

If multiple threads are involved, when the model has multiple inputs, the enqueuing tasks of input data (acltdtEnqueueData) must be in the same thread. When the model has multiple outputs, the dequeuing tasks of output data (acltdtDequeueData) must also be in the same thread.

Sample Code

The following is a code example of key steps for model inference in queue mode. It is for reference only and cannot be directly copied, compiled, and run. After APIs are called, you need to add exception handling branches and record error logs and info logs.

#include "acl/acl.h"
......

//Obtain the run mode of the AI software stack. The API calls for memory allocation and memory copy are different in different run modes.
extern bool g_isDevice;
aclrtRunMode runMode;
aclError ret = aclrtGetRunMode(&runMode);
g_isDevice = (runMode == ACL_DEVICE);

// 1. Load and execute the model.
//The two dots (..) indicate a path relative to the directory of the executable file.
//For example, if the executable file is stored in the out directory, the two dots (..) point to the parent directory of the out directory.
const char* omModelPath = "../model/resnet50.om";

// 1.1 Create an input queue for the model. If the model has multiple inputs, create multiple input queues. The following uses one input as an example.
acltdtQueueAttr *attr = acltdtCreateQueueAttr();
uint32_t *inputQueueList = new (nothrow) uint32_t[num];
int32_t inputNum = 1;

for (int n = 0; n < inputNum; n++) {
        uint32_t inputQid;
        ret = acltdtCreateQueue(attr, &inputQid);
        inputQueueList[n] = inputQid;
    }

// 1.2 Create an output queue for the model. If the model has multiple outputs, create multiple output queues. The following uses one output as an example.
uint32_t *outputQueueList = new (nothrow) uint32_t[num];
int32_t outputNum = 1;

for (int n = 0; n < outputNum; n++) {
        uint32_t outputQid;
        ret = acltdtCreateQueue(attr, &outputQid);
        outputQueueList[n] = outputQid;
    }

// 1.3 Load the model.
uint32_t modelId;
ret= aclmdlLoadFromFileWithQ(modelPath, &modelId, 
                             inputQueueList, inputNum, outputQueueList, outputNum);

// 1.4 Obtain the model description based on the model ID.
aclmdlDesc *modelDesc = aclmdlCreateDesc();
ret = aclmdlGetDesc(modelDesc, modelId);

// 1.5 Obtain the input buffer size of the model. If the model has multiple inputs, obtain the buffer size of each input. The following uses one input as an example.
size_t inputSize = aclmdlGetInputSizeByIndex(modelDesc, 0);

// 1.6 Load the test image data, perform inference, and post-process the inference result data.
string testFile[] = {
        "../data/dog1_1024_683.bin",
        "../data/dog2_1024_683.bin"
};

for (size_t index = 0; index < sizeof(testFile) / sizeof(testFile[0]); ++index) {
        uint32_t devBufferSize;
        void *picDevBuffer = nullptr;
        //Customize the ReadBinFile function, allocate buffer based on the run mode of the AI software stack, and call the function in the C++ standard library that reads the image data into the buffer.
        ret = Utils::ReadBinFile(testFile[index], picDevBuffer, devBufferSize);

        //Pass the model input data to the queue and run model inference. The value -1 indicates that the program is blocked until the input data is completely enqueued.
        ret = acltdtEnqueueData(inputQid, picDevBuffer, devBufferSize, nullptr, 0, -1, 0);
        //Obtain the size of each output.
	size_t dataSize = aclmdlGetOutputSizeByIndex(modelDesc, 0);
	void *data = nullptr;
        size_t retDataSize = 0;
        //Allocate buffer for the model output data.
	if (!g_isDevice) {
            aclError aclRet = aclrtMallocHost(&data, dataSize);
	 } else {
	    aclError aclRet = aclrtMalloc(&data, dataSize, ACL_MEM_MALLOC_HUGE_FIRST);
	 }
         //Wait until the inference execution of the model is complete and obtain the result data from the output buffer. The value -1 indicates that the program is blocked until the inference output data is enqueued.
         ret = acltdtDequeueData(outputQid, data, dataSize, &retDataSize, nullptr, 0, -1);
         //Cast the data in the output buffer to the float type.
         float *outData = NULL;
         outData = reinterpret_cast<float*>(data);
        
        //Print the class indexes of top 5 confidence values.
         map<float, int, greater<float> > resultMap;
         for (int j = 0; j < len / sizeof(float); ++j) {
            resultMap[*outData] = j;
            outData++;
         }
         int cnt = 0;
         for (auto it = resultMap.begin(); it != resultMap.end(); ++it) {
            // print top 5
            if (++cnt > 5) {
               break;
            }
            INFO_LOG("top %d: index[%d] value[%lf]", cnt, it->second, it->first);
         }
         if (!g_isDevice) {
            aclError aclRet = aclrtFreeHost(picDevBuffer);
            aclrtFreeHost(data);
	 } else {
	    aclError aclRet = aclrtFree(picDevBuffer);
            aclrtFree(data); 
	 }
}

// 2. Unload the model and release resources related to model inference.
aclmdlUnload(modelId);
aclmdlDestroyDesc(modelDesc);
acltdtDestroyQueue(inputQid);
acltdtDestroyQueue(outputQid);
acltdtDestroyQueueAttr(attr);

......

Parent topic: Model inference