Documentation

Stream Management

This section describes the processes of creating and destroying a single stream and multiple streams, and synchronization process in the multistreaming scenario.

A stream is a sequence of tasks that execute in order. Different streams, on the other hand, execute their tasks concurrently. A default stream is bound to a default context. If no stream is created explicitly, the default stream can be used. To pass the default stream to any API call, pass NULL directly.

AscendCL provides the following stream management mechanisms.

For details about the synchronization process in the multistreaming scenario, see API Call Sequence and Sample Code for Task Synchronization Between Streams.

Single-threading, Single-streaming Scenario

Following the API calls, add exception handling branches and specify log printing of error and information levels. The following is a code snippet of key steps only, which is not ready to be built or run.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
#include "acl/acl.h"
// ......
int32_t deviceId = 0;

//Specify a compute device.
aclrtSetDevice(deviceId);

//Explicitly create a stream.
aclrtStream stream;
aclrtCreateStream(&stream);

//Pass the stream argument to the task triggering call.
aclrtMemcpyAsync(dstPtr, dstSize, srcPtr, srcSize, ACL_MEMCPY_HOST_TO_DEVICE, stream);
//Call aclrtSynchronizeStream to wait for the stream tasks to complete.
aclrtSynchronizeStream(stream);

//Explicitly destroy the stream after using the stream.
aclrtDestroyStream(stream);
// ......

Single-threading, Multistreaming Scenario

Following the API calls, add exception handling branches and specify log printing of error and information levels. The following is a code snippet of key steps only, which is not ready to be built or run.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
#include "acl/acl.h"
// ......
int32_t deviceId = 0;
uint32_t modelId1 = 0;
uint32_t modelId2 = 1;
aclrtStream stream1;
aclrtStream stream2;

//Specify a compute device.
aclrtSetDevice(deviceId);

//Create stream1.
aclrtCreateStream(&stream1);

//Call the task execution API. For example, the asynchronous model inference tasks are delivered to stream1.
aclmdlDataset *input1;
aclmdlDataset *output1;
aclmdlExecuteAsync(modelId1, input1, output1, stream1);

//Create stream2.
aclrtCreateStream(&stream2);

//Call the task execution API. For example, the asynchronous model inference tasks are delivered to stream2.
aclmdlDataset *input2;
aclmdlDataset *output2;
aclmdlExecuteAsync(modelId2, input1, output2, stream2);

//Synchronize streams.
aclrtSynchronizeStream(stream1);
aclrtSynchronizeStream(stream2);

//Destroy allocations.
aclrtDestroyStream(stream1);
aclrtDestroyStream(stream2);
// ....

Multithreading, Multistreaming Scenario

Following the API calls, add exception handling branches and specify log printing of error and information levels. The following is a code snippet of key steps only, which is not ready to be built or run.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
#include "acl/acl.h"
// ......
void runThread() {
    int32_t deviceId =0;
    //Specify a compute device.
    aclrtSetDevice(deviceId);
    //Explicitly create a stream.
    aclrtStream stream;
    aclrtCreateStream(&stream);
    //Call the task execution API.
    // ....
    //Destroy allocations.
    aclrtDestroyStream(stream);
}

// Create two threads and create a stream in each thread.
std::thread t1(runThread);
std::thread t2(runThread);
//Explicitly call the join functions to ensure that the threads are ended.
t1.join();
t2.join();
Parent topic: Runtime Management