Introduction to Multithreading

Overview

The implementation of multithreading in a multi-device setup involves two key points: (1) How to call the ATB operation API in multithreading. (2) How to create and allocate contexts. This sample uses the basic std::thread class to implement multithreading. Here, the number of threads created is the same as the number of devices. Each thread corresponds to one device. At the beginning, a thread calls the aclrtSetDevice function to bind a device. After the corresponding NPU is bound, the hardware resources required for execution are created, such as the ATB context and device stream.

Figure 1 Illustration of delivering operators via multithreading

ATB Operation API Call by Using Multithreading

Currently, all ATB functional APIs are provided through the Operation. The Setup API and Execute API of the same Operation object have dependencies. Note that the Operation object stores running information. Therefore, the Setup API and Execute API of the same Operation object cannot be executed concurrently.

It is recommended that multiple threads or processes create their own Operation objects. This allows that the Setup API and Execute API of the same Operation object are executed in serial manner, ensuring functional correctness.

Context Creation and Allocation

A context contains 32 × 3 MB graphics memory and device-related resources such as events. In the multithreading or multi-process scenario, one thread or process usually corresponds to one device. Therefore, you are advised to create a context after the device is bound to the thread or process. In addition, a context is bound to a device. The context created using the resources of device 1 cannot be reused on device 2.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

int main()
{
	...
	
    /* Thread creation */

    std::vector<std::thread> threadArray(THREAD_SIZE);
    for (size_t i = 0; i < THREAD_SIZE; i++) {
        Model &model = modelArray.at(i);
        threadArray.at(i) = std::thread([i, &model]{ModelExecute(i, model);}); // Thread creation and function binding
    }
    for (size_t i = 0; i < THREAD_SIZE; i++) {
        threadArray.at(i).join(); // Wait for the child thread to end.
    }

	...
}

/* Create device resources immediately after the thread is started. */
void ModelExecute(uint32_t deviceId, Model &model)
{
    // Initialize the model, and create the required context and stream.
    model.InitResource(deviceId);

    ...
}

void Model::InitResource(uint32_t deviceId)
{
    /* Bind the corresponding device, and then allocate device resources. */
    // Set deviceId.
    deviceId_ = deviceId;
    auto ret = aclrtSetDevice(deviceId_);
    CHECK_RET(ret, "aclrtSetDevice failed. ret: " + std::to_string(ret));

    // Create a context.
    ret = atb::CreateContext(&mode_context_);
    CHECK_RET(ret, "ATB CreateContext failed. ret: " + std::to_string(ret));
	
    // Create a stream.
    ret = aclrtCreateStream(&model_stream_);
    CHECK_RET(ret, "aclrtCreateStream failed. ret: " + std::to_string(ret));

    // Configure the stream.
    mode_context_->SetExecuteStream(model_stream_);
}