Programming Model
Host and Device Programming Model
Host: A host is an x86 or Arm server CPU that connects to one or more devices through a bus. The host utilizes the compute capabilities (for example, NN) provided by the device to implement services.
Device (NPU): refers to a hardware device powered by AI processors. It connects to the host through a bus (such as PCIe and HCCS) and provides, providing compute capabilities (for example, NN). HCCS stands for Huawei Cache Coherence System.
The following figure shows the relationship between the host and the device.
In summary:
- The host and device each possess independent memory spaces. Runtime provides memory allocation APIs for both the host and device sides, along with memory copy APIs to transfer data between them. During programming, developers must distinguish between host and device memory allocations and explicitly call the memory copy APIs to transfer data from the host to the device. This ensures that the device hardware accelerator achieves optimal performance when accessing its local memory.
- Asynchronous parallel execution is used between the host and the device.
After the host delivers a task to the device, the device starts to schedule and execute the delivered task without waiting for the task execution completion. The CPU on the host side and the accelerator on the device side can work concurrently. The API for delivering an asynchronous task usually contains the stream parameter, indicating that the task is delivered to the corresponding stream for execution.
When the host needs to obtain the calculation result of the device, the host must initiate an explicit synchronous API call. The synchronous API blocks the host CPU until the task on the device side is complete.
This asynchronous parallel execution mechanism can effectively hide the processing time of the host or the data transmission delay between the host and the device, improve the throughput, and shorten the end-to-end execution time.
- Execution mode of a task delivered to a stream in asynchronous mode.
Tasks in streams are executed in sequence, and tasks in different streams are executed concurrently. As shown in the following figure, the Kernel1, Kernel2, and Kernel3 tasks are started in sequence on the host. The execution sequence is as follows:
- Kernel1 and Kernel3 are in the same stream. Therefore, Kernel3 can be executed only after Kernel1 is executed.
- Kernel 2 is not in the same stream as Kernel 1 and Kernel 3. Therefore, Kernel 2 can be executed in parallel with Kernel 1 and Kernel 3.
Main Programming Concepts of Runtime
- Host is the abstraction of Runtime on the host.
- Device represents the abstraction of the hardware device where the AI processor is installed. Typically, one host maps to N devices. The user application can call ACL APIs such as aclrtSetDevice to specify the compute device.
- The Context is the logical running environment of the device. The relationship between the context and the device is N:1. That is, each context belongs to a unique device. A context manages the lifetime of its runtime resource objects (including streams, events, and notifies, but excluding memory). Each unique context has isolated objects, and they cannot be synchronized with one another. Runtime errors are also isolated at the context level.
- Stream is a logical task execution queue provided by the device. Tasks can be asynchronously added to a stream. Tasks in the same stream are executed in FIFO mode. The relationship between streams and contexts is N:1. A stream must belong to a unique context.
- Task: execution tasks that can be added to a stream. The tasks can be classified into computing tasks, memory copy tasks, and event synchronization tasks. The relationship between tasks and streams is N:1. A task is added to a unique stream.
The following figure shows the relationship between the device, context, and stream.
Thread-Context Association
Most APIs of Runtime do not have the device ID parameter because the device on which these APIs function is obtained from the context associated with the calling thread. Therefore, when the host thread calls the Runtime API, the following requirements must be met:
- Runtime APIs can be correctly called only after a thread (CPU thread on the host) is associated with a context.
- A thread can be associated with only one context at a time.
- Applications can explicitly create contexts to meet the service requirements of running resource isolation. In this scenario, the context in the same process can be viewed by all threads. Threads can use aclrtGetCurrentContext and aclrtSetCurrentContext to switch the context.
The following example describes the association and switching between the thread and context when the thread calls the context-related APIs. The example is for reference only and cannot be directly copied and run.
# Initially, the thread is not associated with any context. acl.init() acl.rt.set_device(0) # acl.rt.set_device creates the default context and associates the thread with the default context. # Context associated with the thread: default context ctx1, _ = acl.rt.create_context(0) # Device 0 explicitly creates ctx1 and associates the thread with ctx1. # Context associated with the thread: ctx1 current_ctx, _ = acl.rt.get_context() #: obtains the context associated with the current thread. In this case, current_ctx = = ctx1 is returned. # Context associated with the thread: ctx1 ctx2, _ = acl.rt.create_context(0) # Device 0 explicitly creates ctx2 and associates the thread with ctx2. # Context associated with the thread: ctx2 The acl.rt.set_context(current_ctx) # switches the context. Because current_ctx is set to ctx1, the thread is associated with ctx1. # Context associated with the thread: ctx1 acl.rt.set_context(current_ctx) #: switches the context. # Context associated with the thread: ctx2 ..... The current thread of acl.rt.destroy_context(ctx2) # is associated with ctx2. When ctx2 is destroyed, the thread is also disassociated from ctx2. # Context associated with the thread: NA # ctx2 has been destroyed. Switch to another ctx (for example, ctx1). acl.rt.set_context(ctx1) # Context associated with the thread: ctx1 ..... The current thread of acl.rt.destroy_context(ctx1); # is associated with ctx1. When ctx1 is destroyed, the thread is also disassociated from ctx1. # Context associated with the thread: NA acl.rt.reset_device_force(0) acl.finalize()
Application Scenarios of Default Contexts and Default Streams
- A context and a stream must exist before any requests are delivered to the device, and these can be created either explicitly or implicitly (default). The implicitly created context and stream are the default context and stream.
To pass the default stream to any API call, pass 0 directly.
- The default context does not allow users to perform the acl.rt.get_context, acl.rt.set_context, or acl.rt.destroy_context operation.
- An implicitly created context or stream is applicable to simple apps where only one compute device is needed. For a multithreaded app, you are advised to use the explicitly created context and stream.
The sample code is as follows, which is for reference only. Do not directly copy and run the code.
# ... ret = acl.init(config_path) ret = acl.rt.set_device(device_id) # In the default context, a default stream is created, and the stream is available in the current thread. # ... ret = acl.op.execute_v2(op1, input_desc, inputs, output_desc, outputs, attr, 0) # 0 indicates that op1 is executed in the default stream. ret = acl.op.execute_v2(op2, input_desc, inputs, output_desc, outputs, attr, 0) # 0 indicates that op2 is executed in the default stream. ret = acl.rt.synchronize_stream(0) # Output the result as required when all computing tasks (op1 and op2 execution tasks) are complete. # ... ret = acl.rt.reset_device(device_id) # Reset device 0. The lifetime of the corresponding default context and default stream end.