[object Object]

Tensor parallelism (TP) is a model parallelism strategy that splits tensors (such as weight matrices and activation values) among multiple devices (such as NPUs) to implement distributed model inference.

[object Object]

• The Atlas 800I A2 inference server and Atlas 800I A3 SuperPoD server support this feature.
• DeepSeek-V3 and DeepSeek-R1 support local TP splitting of the LmHead matrix, local TP splitting of the O project matrix, and TP greater than 1.
• Prefill-decode disaggregation scenarios with distributed decode nodes support local TP splitting of the LmHead and O project matrices, which reduces the matrix computation time and inference latency.
• In prefill-decode disaggregation scenarios with distributed, low-latency decode nodes, if TP exceeds 1, TP splitting of MLA is supported, which reduces the decode inference latency in small-batch and low-latency scenarios.
• If [object Object] exceeds 1, this feature cannot be enabled together with local TP splitting of the O project matrix, and you are not advised to enable this feature together with local TP splitting of the LmHead matrix.

[object Object]

describes the parameters required for enabling local TP splitting of the LmHead matrix.

Table 1 Parameter for local TP splitting of the LmHead matrix: [object Object] in [object Object] [object Object][object Object]

[object Object]undefined

describes the parameters required for enabling local TP splitting of the O project matrix.

Table 2 Parameter for local TP splitting of the O project matrix: [object Object] in [object Object] [object Object][object Object]

[object Object]undefined[object Object]

1. Open the [object Object] file of the server.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

2. Set serving parameters. Add parameters to the [object Object] file of the server based on and . For details about the serving parameters, see . The following is an example of parameter configuration.

   The following uses the DeepSeek-R1 model as an example. In addition, enabling TP splitting and disabling local TP splitting of the LmHead and O project matrices are used as examples.

   [object Object]

3. Start the service.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]