Data Parallel

Data parallelism (DP) splits inference requests into multiple batches and allocates them to different compute devices for parallel processing. These devices process different batches of data in parallel, and then merge the results.

Scenario

When the graphics memory is sufficient, the data parallel feature can be enabled to improve the throughput.

Constraints

• The Atlas 800I A2 inference server and Atlas 800I A3 SuperPoD server support this feature.
• The Attention and MLP modules of all models support this feature.
• DP can be used together with tensor parallelism in the same module.

Description

Table 1 describes the supplementary parameters required for enabling the data parallel feature.

Table 1 ModelConfig in ModelDeployConfig

Inference

CANN and MindIE have been installed in the environment. For details, see MindIE Installation Guide.

1. Set environment variables for optimizing graphics memory allocation.

   export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
   export ATB_WORKSPACE_MEM_ALLOC_ALG_TYPE=3

2. Open the config.json file of the server.

   • Installation using the .whl package:

   cd {MindIE_installation_directory}/mindie_llm/
   vi conf/config.json

   • Installation using the .run package:

   cd {MindIE_installation_directory}/latest/mindie-service
   vi conf/config.json

3. Set serving parameters. Add parameters to the config.json file of the server according to Table 1. For details about the serving parameters, see Configuration Parameters (Serving). The following is an example of parameter settings.

   "ModelConfig" : [
       {
           "modelInstanceType" : "Standard",
           "modelName" : "deepseekv2",
           "modelWeightPath" : "/home/data/DeepSeek-V2-Chat-W8A8-BF16/",
           "worldSize" : 8,
           "cpuMemSize" : 5,
           "npuMemSize" : 1,
           "backendType" : "atb",
           "trustRemoteCode" : false,
           "tp": 1,
           "dp": 8,
           "cp": 1,
           "sp": 1
       }
   ]

   In the preceding parameter settings, eight devices are used for inference, the Attention module uses DP, and the MoE model uses tensor parallelism.

4. Start the service.

   • Installation using the .whl package:

   mindie_llm_server

   • Installation using the .run package:

   ./bin/mindieservice_daemon

5. Send an inference request. For details, see "Cluster Management Components" > "Coordinator" > "RESTful APIs" > "User-Side APIs" > "OpenAI Inference APIs" in MindIE Motor Development Guide.