Task Update

After the tasks in the stream are captured and temporarily stored in the model, you can update the tasks (including the tasks themselves and task parameter information) in either of the following ways:

1. Method 1: Use the task to be updated for demarcation. Capture the tasks before and after that task between aclmdlRICaptureBegin and aclmdlRICaptureEnd, and temporarily store the tasks in different models for separate execution.

   This method is suitable for scenarios where a large number of tasks need to be updated (for example, a model has inputs of two types of shapes). The API call logic is simple, but the number of models that temporarily store captured tasks is increasing. If the number of models exceeds the hardware resource limit, an error is reported.

   The following figure shows the basic process of this method.

   

2. Method 2: Deliver the tasks to be captured in the main stream between the aclmdlRICaptureBegin and aclmdlRICaptureEnd. Use aclmdlRICaptureTaskGrpBegin and aclmdlRICaptureTaskGrpEnd to mark the tasks to be captured in a task group, return the handle of the task group, and update the tasks between aclmdlRICaptureTaskUpdateBegin and aclmdlRICaptureTaskUpdateEnd.

   This method is suitable for the scenario where a small number of single-operator call tasks need to be updated. You can update tasks and then execute the tasks in the model instance in sequence, or update the tasks and execute other tasks in the model instance concurrently. However, updating tasks is more time-consuming than delivering tasks separately. In addition, there are some restrictions: The number and types of tasks between aclmdlRICaptureTaskGrpBegin and aclmdlRICaptureTaskGrpEnd must be the same as those between aclmdlRICaptureTaskUpdateBegin and aclmdlRICaptureTaskUpdateEnd. In the cross-stream task capture scenario, tasks cannot be delivered to streams in other capture states at the same time between aclmdlRICaptureTaskGrpBegin and aclmdlRICaptureTaskGrpEnd. A task group is similar to a critical resource and does not support concurrent update of multiple threads and streams. Otherwise, the update result may be unexpected.

   • The following figure shows the process of updating tasks and then executing tasks in the aclmdlRI instance in sequence.

     

   • The following figure shows the process of updating tasks and executing other tasks concurrently.

     If a large number of tasks exist in the running instance of the model, you can use an external event to update tasks and execute other tasks concurrently to improve performance. In addition, you need to create a stream (UpdateStream) for updating tasks. The external event refers to the event that is created through aclrtCreateEventWithFlag, with the flag set to ACL_EVENT_EXTERNAL. This type of event cannot be used for cross-stream task capture, and their specifications are limited. For this reason, you need to properly reuse this type of event. After creating an external event, deliver an update task in UpdateStream, and then call aclrtRecordEvent to deliver an Event Record task. Then, in the main stream, call aclrtStreamWaitEvent to deliver an Event Wait task before the task to be updated, to wait for the task update in UpdateStream to complete. Finally, in the main stream, call aclrtStreamWaitEvent and then aclrtResetEvent to reset the external event.

     

     The following is the sample code for updating the input parameters of the aclnnAdd operator in the concurrent task execution scenario:

     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199

     #include <stdio.h> #include <vector> #include "acl/acl.h" #include "aclnnop/aclnn_add.h" #define ACL_LOG(fmt, args...) fprintf(stdout, "[INFO] " fmt "\n", ##args) int64_t GetShapeSize(const std::vector<int64_t> &shape) { int64_t shape_size = 1; for (auto i : shape) { shape_size *= i; } return shape_size; } int CreateAclTensor(const std::vector<int64_t> &shape, void **deviceAddr, aclDataType dataType, aclTensor **tensor) { auto size = GetShapeSize(shape) * sizeof(float); // Allocate the device memory. auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST); // Compute the strides of the contiguous tensor. std::vector<int64_t> strides(shape.size(), 1); for (int64_t i = shape.size() - 2; i >= 0; i--) { strides[i] = shape[i + 1] * strides[i + 1]; } // Call aclCreateTensor to create an aclTensor. *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(), *deviceAddr); return 0; } int main() { int devID = 0; void *self_d = nullptr; void *other_d = nullptr; void *out_d = nullptr; void *outtmp_d = nullptr; aclTensor *self = nullptr; aclTensor *other = nullptr; aclScalar *alpha = nullptr; aclScalar *updatealpha = nullptr; aclTensor *out = nullptr; aclTensor *outtmp = nullptr; /* aclnnAdd: self = self + other * alpha */ float *self_h = nullptr; float *other_h = nullptr; std::vector<int64_t> shape = {4, 2}; float *out_h = nullptr; float alphaValue = 1.1f; float updatealphaValue = 5.5f; uint64_t workspaceSize = 0; uint64_t workspaceSize1 = 0; uint64_t workspaceSize2 = 0; aclOpExecutor *executor2; aclOpExecutor *executor; aclOpExecutor *executor1; auto size = GetShapeSize(shape); // Perform initialization. aclInit(NULL); // Specify the compute device. aclrtSetDevice(devID); // Prepare the input and output parameters of the aclnnAdd operator. CreateAclTensor(shape, &self_d, aclDataType::ACL_FLOAT, &self); CreateAclTensor(shape, &other_d, aclDataType::ACL_FLOAT, &other); alpha = aclCreateScalar(&alphaValue, aclDataType::ACL_FLOAT); updatealpha = aclCreateScalar(&updatealphaValue, aclDataType::ACL_FLOAT); CreateAclTensor(shape, &out_d, aclDataType::ACL_FLOAT, &out); CreateAclTensor(shape, &outtmp_d, aclDataType::ACL_FLOAT, &outtmp); // Call the first-phase API of the aclnnAdd operator to obtain the workspace size required for operator computation and the executor that contains the operator computation process. // If aclnnAdd is called multiple times, the first-phase API needs to be called multiple times to obtain different aclOpExecutors. // outtmp = self + alpha * other // Before the update: out = outtmp + alpha * other After the update: out = outtmp + updatealpha * other aclnnAddGetWorkspaceSize(self, other, alpha, outtmp, &workspaceSize, &executor); void *workspaceAddr = nullptr; if (workspaceSize > 0) { aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); } // Before the update: out = outtmp + alpha * other aclnnAddGetWorkspaceSize(outtmp, other, alpha, out, &workspaceSize1, &executor1); void *workspaceAddr1 = nullptr; if (workspaceSize1 > 0) { aclrtMalloc(&workspaceAddr1, workspaceSize1, ACL_MEM_MALLOC_HUGE_FIRST); } // After the update: out = outtmp + updatealpha * other aclnnAddGetWorkspaceSize(outtmp, other, updatealpha, out, &workspaceSize2, &executor2); void *workspaceAddr2 = nullptr; if (workspaceSize2 > 0) { aclrtMalloc(&workspaceAddr2, workspaceSize2, ACL_MEM_MALLOC_HUGE_FIRST); } // Allocate page-locked memory by using aclrtMallocHost. aclrtMallocHost((void **)&self_h, size * sizeof(float)); aclrtMallocHost((void **)&other_h, size * sizeof(float)); aclrtMallocHost((void **)&out_h, size * sizeof(float)); for (int i = 0; i < 8; i++) { self_h[i] = static_cast<float>(0); other_h[i] = static_cast<float>(1); out_h[i] = static_cast<float>(0); } aclmdlRI modelRI; aclrtStream stream1; aclrtCreateStream(&stream1); aclrtEvent event; // Create an external event. aclrtCreateEventWithFlag(&event, ACL_EVENT_EXTERNAL); // ====== ==Start the capture task.======== aclmdlRICaptureBegin(stream1, ACL_MODEL_RI_CAPTURE_MODE_GLOBAL); // Asynchronous copy, which is used to copy the input data of the self parameter of the aclnnAdd operator from the host to the device aclrtMemcpyAsync(self_d, size * sizeof(float), self_h, size * sizeof(float), ACL_MEMCPY_HOST_TO_DEVICE, stream1); // Asynchronous copy, which is used to copy the input data of the other parameter of the aclnnAdd operator from the host to the device aclrtMemcpyAsync(other_d, size * sizeof(float), other_h, size * sizeof(float), ACL_MEMCPY_HOST_TO_DEVICE, stream1); // Execute the aclnnAdd operator. aclnnAdd(workspaceAddr, workspaceSize, executor, stream1); // Deliver an Event Wait task in the main stream (stream 1) to wait for the update task to complete. aclrtStreamWaitEvent(stream1, event); aclrtResetEvent(event, stream1); aclrtTaskGrp handle; // Mark the task to be updated. aclmdlRICaptureTaskGrpBegin(stream1); aclnnAdd(workspaceAddr1, workspaceSize1, executor1, stream1); aclmdlRICaptureTaskGrpEnd(stream1, &handle); // Asynchronous copy, which is used to copy the output data of the operator from the device to the host aclrtMemcpyAsync(out_h, size * sizeof(float), out_d, size * sizeof(float), ACL_MEMCPY_DEVICE_TO_HOST, stream1); // ====== ==End the capture task.======== aclmdlRICaptureEnd(stream1, &modelRI); aclrtStream updateStream; aclrtCreateStream(&updateStream); for (int i = 0; i < 2; i++) { ACL_LOG("execute model, loop: %d", i); aclmdlRIExecuteAsync(modelRI, stream1); // Start the update task and update the alpha parameter of the aclnnAdd operator to updatealpha. aclmdlRICaptureTaskUpdateBegin(updateStream, handle); if (i == 1) { aclnnAdd(workspaceAddr2, workspaceSize2, executor2, updateStream); ACL_LOG("update alpha value of aclnnAdd"); } aclmdlRICaptureTaskUpdateEnd(updateStream); // After the update task is complete, deliver an Event Record task in UpdateStream to notify the main stream (stream 1) to continue with the tasks following the Event Wait task. aclrtRecordEvent(event, updateStream); aclrtSynchronizeStream(updateStream); aclrtSynchronizeStream(stream1); ACL_LOG("%f %f %f %f %f %f %f %f\n", out_h[0], out_h[1], out_h[2], out_h[3], out_h[4], out_h[5], out_h[6], out_h[7]); } // Free resources. aclmdlRIDestroy(modelRI); aclDestroyTensor(self); aclDestroyTensor(other); aclDestroyTensor(out); aclDestroyTensor(outtmp); aclDestroyScalar(alpha); aclDestroyScalar(updatealpha); aclrtFree(self_d); aclrtFree(other_d); aclrtFree(out_d); aclrtFree(outtmp_d); aclrtDestroyStream(stream1); aclrtDestroyStream(updateStream); aclrtDestroyEvent(event); if (workspaceAddr != nullptr) { aclrtFree(workspaceAddr); } if (workspaceAddr1 != nullptr) { aclrtFree(workspaceAddr1); } if (workspaceAddr2 != nullptr) { aclrtFree(workspaceAddr2); } // Free the resources of the compute device. aclrtResetDevice(devID); // Perform deinitialization. aclFinalize(); }