Weight Update (User-managed Memory)

API Call Sequence

In the weight update scenario, you can call the following APIs to dynamically update the weight during model execution after the model is built:

Build and save the model based on graph build APIs. The model contains multiple graphs, such as the inference graph, variable initialization graph, and variable update graph.
In this section, aclgrphBundleBuildModel is called to build the model, and aclgrphBundleSaveModel is called to save the model. For details about the APIs, see aclgrphBundleBuildModel and aclgrphBundleSaveModel.

Weight initialization is optional. You can determine whether to include the weight initialization graph based on the service scenario. If the weight initialization graph is not included, the device memory required for model loading can be saved.
Call aclmdlBundleQueryInfoFromFile or aclmdlBundleQueryInfoFromMem to obtain the model description.
Call aclmdlBundleGetQueryModelNum to obtain the total number of graphs in the bundle model.
Call aclmdlBundleGetVarWeightSize and aclmdlBundleGetSize to obtain memory information required by the model, including the size of the weight that can be updated and the size of the workspace and weight memory required by each graph.
Call aclmdlBundleInitFromFile or aclmdlBundleInitFromMem to initialize the model.
Call aclmdlBundleLoadModelWithMem multiple times based on the index to load the graph. The allocated workspace and weight memory can be transferred to obtain the model ID of the corresponding graph.
Call the model execution API (for example, aclmdlExecute) to execute the weight initialization graph based on the model ID of the graph.
To update the weight, call aclmdlSetDatasetTensorDesc to set the tensor description of the graph before updating the weight.
Update the graph based on the model ID of the weight update graph and call the model execution API (for example, aclmdlExecute) to update the weight graph.
Call the model execution API (for example, aclmdlExecute) to execute the inference graph based on the model ID of the inference graph.
After the inference is complete, call aclmdlBundleUnloadModel to unload the graph, and then call aclmdlBundleUnload to unload the model.

Sample Code

The following is a code snippet of key steps for model inference. It is for reference only. Do not copy, build, or run it directly. After APIs are called, you need to add exception handling branches and record error logs and info logs.

       
        
          
          
            // 1. Initialize resources.
aclInit(nullptr);
aclrtSetDevice(0);

// 2. Query the memory information of the model built based on graph construction APIs. The model contains the inference graph, weight initialization graph, and weight update graph. The model file bundle.om is used as an example.
aclmdlBundleQueryInfo *query_info = aclmdlBundleCreateQueryInfo();
aclmdlBundleQueryInfoFromFile("./bundle.om", query_info);

// 3. Obtain the total number of graphs in the model and memory information based on the query result.
size_t modelNum = 0;
aclmdlBundleGetQueryModelNum(query_info, &modelNum);
// (Optional) Obtain the size of the updateable weight required by the model and allocate memory. If no memory is allocated, the system allocates memory internally.
size_t variableWeightSize = 0;
aclmdlBundleGetVarWeightSize(query_info, &variableWeightSize);
void *var_p = nullptr;
aclrtMalloc(&var_p, variableWeightSize, ACL_MEM_MALLOC_NORMAL_ONLY);
// Assume that modelNum is set to 3. The input parameters of the aclgrphBundleBuildModel API are three graphs with fixed IDs, which are 0, 1, and 2.
// (Optional) Obtain the sizes of the workspace and weight memory required by the inference graph with an index of 0 and allocate memory. If no memory is allocated, the system allocates memory internally.
size_t inferWorkSize, inferConstWeightSize;
aclmdlBundleGetSize(query_info, 0, &inferWorkSize, &inferConstWeightSize);
void *inferWorkPtr = nullptr;
aclrtMalloc(&inferWorkPtr, inferWorkSize, ACL_MEM_MALLOC_NORMAL_ONLY);
void *inferConstWeightPtr = nullptr;
aclrtMalloc(&inferConstWeightPtr, inferConstWeightSize, ACL_MEM_MALLOC_NORMAL_ONLY);

// 4: Initialize bundle.om.
uint32_t bundle_id = 0;
// If the system allocates memory internally, pass nullptr to var_p and 0 to variableWeightSize.
aclmdlBundleInitFromFile("./bundle.om", var_p, variableWeightSize, &bundle_id);

// 5. Load the graph as required based on the index.
uint32_t infer_id= 0;
// If the system allocates memory internally, use aclmdlBundleLoadModelWithMem(bundle_id, 0, nullptr, 0, nullptr, 0, &init_id).
aclmdlBundleLoadModelWithMem(bundle_id, 0, inferWorkPtr, inferWorkSize,inferConstWeightPtr, inferConstWeightSize, &infer_id);
uint32_t init_id= 0;
aclmdlBundleLoadModelWithMem(bundle_id, 1, nullptr, 0, nullptr, 0, &init_id);
uint32_t update_id= 0;
aclmdlBundleLoadModelWithMem(bundle_id, 2, nullptr, 0, nullptr, 0, &update_id);

// If the weight does not need to be updated, execute the weight initialization graph and inference graph.
// 6. Execute the weight initialization graph. For details about how to prepare the model input and output, see the sample code in other sections about inference features under "Model Inference".
aclmdlExecute(init_id, init_mdl_input, init_mdl_output);

// 7. Execute the inference graph. For details about how to prepare the model input and output, see the sample code in other sections about inference features under "Model Inference".
aclmdlExecute(infer_id, infer_mdl_input, infer_mdl_output);

// If a weight needs to be updated, update the weight before executing the inference graph.
// 8. Execute the weight update graph.
// If a weight does not need to be updated, for example, the 0th weight, the shape can be passed as an empty tensor, but the device memory must be valid.
size_t no_need_refresh_index = 0;
std::vector<int64_t> dims{0};
// If the elements in the dims array are 0, the tensor is empty.
auto tensorDesc = aclCreateTensorDesc(ACL_FLOAT, dims.size(), dims.data(), ACL_FORMAT_ND);
aclmdlSetDatasetTensorDesc(update_mdl_input, tensorDesc, no_need_refresh_index);

// 9. The following is an example of updating the first weight.
size_t need_refresh_index = 1;
std::vector<int64_t> dims{1, 3, 224, 224};
auto tensorDesc = aclCreateTensorDesc(ACL_FLOAT, dims.size(), dims.data(), ACL_FORMAT_ND);
aclmdlSetDatasetTensorDesc(update_mdl_input, tensorDesc, need_refresh_index);

// 10. Execute the weight update graph. For details about how to prepare the model input and output, see the sample code in other sections about inference features under "Model Inference".
aclmdlExecute(update_id, update_mdl_input, update_mdl_output);

// 11. Execute the inference graph. For details about how to prepare the model input and output, see the sample code in other sections about inference features under "Model Inference".
aclmdlExecute(infer_id, infer_mdl_input, infer_mdl_output);

// 12. Unload the model.
aclmdlBundleUnloadModel(bundle_id, infer_id);
aclmdlBundleUnloadModel(bundle_id, init_id);
aclmdlBundleUnloadModel(bundle_id, update_id);
aclmdlBundleUnload(bundle_id);

// 13. Destroy allocations.
aclrtResetDevice(0);
aclFinalize();

           

         

       
      

Parent topic: Weight Update