Stream Management

This section describes the processes of creating and destroying a single stream and multiple streams, and the synchronization process in the multistreaming scenario.

A stream is a sequence of tasks that execute in order. Different streams, on the other hand, execute their tasks concurrently. A default stream is bound to a default context. If no stream is created explicitly, the default stream can be used. To pass the default stream to any API call, pass NULL directly.

The following stream management mechanisms are now available:

Single-Threading, Single-Streaming Scenario
Single-Threading, Multistreaming Scenario
Multithreading, Multistreaming Scenario

For details about the synchronization process in the multistreaming scenario, see Synchronous Wait of Tasks Between Streams (Implemented Using Events).

Single-Threading, Single-Streaming Scenario

Following the API calls, add exception handling branches and specify log printing of error and information levels. The following is a code snippet of key steps only, which is not ready to be built or run.

#include "acl/acl.h"
// ......
int32_t deviceId = 0;

//Specify a compute device.
aclrtSetDevice(deviceId);

//Explicitly create a stream.
aclrtStream stream;
aclrtCreateStream(&stream);

//Pass the stream argument to the task triggering call.
aclrtMemcpyAsync(dstPtr, dstSize, srcPtr, srcSize, ACL_MEMCPY_HOST_TO_DEVICE, stream);
//Call aclrtSynchronizeStream to block the app until all tasks in the specified stream are complete.
aclrtSynchronizeStream(stream);

//Explicitly destroy the stream after using the stream.
aclrtDestroyStream(stream);
// ......

Single-Threading, Multistreaming Scenario

#include "acl/acl.h"
// ......
int32_t deviceId = 0;
uint32_t modelId1 = 0;
uint32_t modelId2 = 1;
aclrtStream stream1;
aclrtStream stream2;

//Specify a compute device.
aclrtSetDevice(deviceId);

//Create stream1.
aclrtCreateStream(&stream1);

//Call the task execution API. For example, the asynchronous model inference tasks are delivered to stream1.
aclmdlDataset *input1;
aclmdlDataset *output1;
aclmdlExecuteAsync(modelId1, input1, output1, stream1);

//Create stream2.
aclrtCreateStream(&stream2);

//Call the task execution API. For example, the asynchronous model inference tasks are delivered to stream2.
aclmdlDataset *input2;
aclmdlDataset *output2;
aclmdlExecuteAsync(modelId2, input2, output2, stream2);

//Synchronize streams.
aclrtSynchronizeStream(stream1);
aclrtSynchronizeStream(stream2);

//Destroy allocations.
aclrtDestroyStream(stream1);
aclrtDestroyStream(stream2);
// ....

Multithreading, Multistreaming Scenario

#include "acl/acl.h"
// ......
void runThread() {
    int32_t deviceId =0;
    //Specify a compute device.
    aclrtSetDevice(deviceId);
    //Explicitly create a stream.
    aclrtStream stream;
    aclrtCreateStream(&stream);
    //Call the task execution API.
    // ....
    //Destroy allocations.
    aclrtDestroyStream(stream);
}

//Create two threads. Each thread internally creates a stream.
std::thread t1(runThread);
std::thread t2(runThread);
//Explicitly call the join functions to ensure that the threads are ended.
t1.join();
t2.join();

Parent topic: Runtime Management