Multi-Device Programming Overview

Multi-device programming enables you to leverage resources, such as the comprehensive performance and memory, of multiple neural-network processing units (NPUs) to achieve performance beyond that of a single NPU. Generally, each physical NPU is abstracted as a device in the Runtime programming interface. Task delivery requires the support of the context in the device. Therefore, multi-device programming needs to manage multiple devices and their corresponding contexts.

Some common multi-NPU programming methods are as follows:

• A single host thread drives multiple NPUs.
• Multiple host threads are used, and each drives its own NPU.
• Multiple single-thread host processes are used, and each drives its own NPU.
• Multiple host processes are used, each process contains multiple threads, and each thread drives its own NPU.