[object Object]

Data parallelism (DP) splits inference requests into multiple batches and allocates them to different compute devices for parallel processing. These devices process different batches of data in parallel, and then merge the results.

[object Object]

When the graphics memory is sufficient, the data parallel feature can be enabled to improve the throughput.

[object Object]

• The Atlas 800I A2 inference server and Atlas 800I A3 SuperPoD server support this feature.
• The Attention and MLP modules of all models support this feature.
• DP can be used together with tensor parallelism in the same module.

[object Object]

describes the supplementary parameters required for enabling the data parallel feature.

Table 1 [object Object] in [object Object] [object Object][object Object]

[object Object]undefined

[object Object]

[object Object]

CANN and MindIE have been installed in the environment. For details, see MindIE Installation Guide.

1. Set environment variables for optimizing graphics memory allocation.

   [object Object]

2. Open the [object Object] file of the server.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

3. Set serving parameters. Add parameters to the [object Object] file of the server according to . For details about the serving parameters, see . The following is an example of parameter settings.

   [object Object]

   In the preceding parameter settings, eight devices are used for inference, the Attention module uses DP, and the MoE model does not use tensor parallelism.

4. Start the service.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

5. Send an inference request. For details, see "Cluster Management Components" > "Coordinator" > "RESTful APIs" > "User-Side APIs" > "OpenAI Inference APIs" in MindIE Motor Development Guide.