[object Object]

Note: This API will be deprecated in later versions. Use the latest API instead.

[object Object][object Object]undefined[object Object]

• Adapts to the [object Object] operator in the decode ([object Object]) and prefill ([object Object]) inference scenarios. Compared with [object Object], this API introduces new parameters: [object Object], [object Object], and [object Object].

  Note: KV cache specific to the decode scenario: KV cache is a common technology for optimizing the inference performance of foundation models. During sampling, the transformer model uses the given prompt/context as the initial input for inference (parallel processing supported), and then generates additional tokens one by one to improve the generated sequence (reflecting the auto-regressive property of the model). The transformer performs the self-attention operation during sampling. Therefore, KV vectors need to be extracted for each item (regardless of the prompt/context or generated token) in the current sequence. These vectors are stored in a matrix called KV cache.

• Formula

  Self-attention constructs an attention model by leveraging the relationships within the input samples. The principle assumes there is an input sample sequence $x$ of length $n$, where each element of $x$ is a $d$-dimensional vector. Each $d$-dimensional vector can be regarded as a token embedding. Such a sequence is transformed by three weight matrices to obtain three $n × d$ matrices.

  The computation formula for self-attention is generally defined as follows, where $Q$, $K$, and $V$ are key attribute elements of the input sample, obtained through spatial transformation and unified into a single feature space. "Attention" in the formula and operator name is an abbreviation for "self-attention."

  $$Attention(Q,K,V)=Score(Q,K)V$$

  The [object Object] function in this operator employs the [object Object] function. The self-attention calculation formula is as follows:

  $$Attention(Q,K,V)=Softmax(\frac{QK^T}{\sqrt{d}})V$$

  The product of $Q$ and $K^T$ represents the attention to the input $x$. To prevent this value from becoming excessively large, it is typically scaled by dividing by the square root of $d$, followed by row-wise softmax normalization. The result is then multiplied by $V$ to produce an $n × d$ matrix.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation

  • [object Object] defaults to a deterministic implementation.

• When this API is used together with PyTorch, ensure that the CANN package versions match the PyTorch package versions.

• Processing logic for a null input parameter: The operator checks whether [object Object] is a null pointer. If so, an error is reported. If [object Object] is not an empty tensor but [object Object] and [object Object] are empty tensors (that is, [object Object] is 0), [object Object] is filled with all zeros. When attentionOut is an empty tensor, the AscendCLNN framework will process it. For other input parameters which support the passing of [object Object] as described in the preceding parameter description, no processing is performed when they are null pointers.

• The shapes of the tensors corresponding to [object Object] and [object Object] must be identical. In non-contiguous scenarios, the batch size in the tensor lists of [object Object] and [object Object] can only be [object Object], the number of elements must be equal to the batch size ([object Object]) of [object Object], and the [object Object] and [object Object] dimensions must be the same. Due to the tensor list restrictions, [object Object] cannot be greater than [object Object] in non-contiguous scenarios.

• When the data type of [object Object] is INT8 or UINT8, the value in the tensor must be [object Object] or [object Object].

• [object Object][object Object]Restrictions on the number of input parameters and input and output data formats related to INT8 quantization:

  • If both the input and output are of the INT8 type, the input parameters [object Object], [object Object], [object Object], and [object Object] must exist at the same time. [object Object] is optional and defaults to [object Object] if not passed.
  • If the input is of the INT8 type and the output is of the FLOAT16 type, the input parameters [object Object], [object Object], and [object Object] must exist at the same time. If the input parameter [object Object] or [object Object] exists (not [object Object]), an error is reported and returned.
  • If the input is of the FLOAT16 or BFLOAT16 type and the output is of the INT8 type, the input parameter [object Object] must exist, and [object Object] is optional ([object Object] is used if no value is passed). If the input parameter [object Object], [object Object], or [object Object] exists (not [object Object]), an error is reported and returned.
  • The input parameters [object Object] and [object Object] support both the per-tensor and per-channel data formats and the FLOAT32 and BFLOAT16 data types. If [object Object] is passed, ensure that its type and shape are consistent with those of [object Object]. If the input is of the BFLOAT16 type, both FLOAT32 and BFLOAT16 are supported. Otherwise, only FLOAT32 is supported. In per-channel format, when the output layout is [object Object], the product of all dimensions of [object Object] must be equal to [object Object]. For other layouts, the product must be equal to [object Object] × [object Object]. (When the output layout is [object Object]BSH[object Object], it is recommended that the shape of [object Object]quantScale2[object Object] be set to [object Object][1, 1, H][object Object] or [object Object][H][object Object]. When the output layout is [object Object]BNSD[object Object], it is recommended that the shape of [object Object]quantScale2[object Object] be set to [object Object][1, N, 1, D][object Object] or [object Object][N, D][object Object]. When the output layout is [object Object]BSND[object Object], it is recommended that the shape of [object Object]quantScale2[object Object] be set to [object Object][1, 1, N, D][object Object] or [object Object][N, D][object Object].)
  • inputLayout supports only BSH, BNSD, BSND, and BNSD_BSND.

• [object Object][object Object]Constraints on the fake-quantization parameters [object Object] and [object Object]:

  • Only the fake-quantization scenario where [object Object] is INT8 is supported.
  • Per-channel mode: The shapes of the two parameters can be [object Object], [object Object], or [object Object], where [object Object] is [object Object]. The data type is the same as that of [object Object], and [object Object] is set to [object Object].
  • Per-tensor mode: The shapes of the two parameters are [object Object], the data type is the same as that of [object Object], and [object Object] is set to [object Object].
  • Per-token mode: The shapes of the two parameters are [object Object], the data type is fixed at FLOAT32, and [object Object] is set to [object Object].
  • In asymmetric quantization mode, both [object Object] and [object Object] must be present.
  • In symmetric quantization mode, [object Object] can be [object Object]. If [object Object] is [object Object], symmetric quantization is performed. Otherwise, asymmetric quantization is performed.

• Restrictions on the input parameters [object Object], [object Object], and [object Object] when the layout is [object Object], [object Object], or [object Object]:

  • Both [object Object] and [object Object] must be passed, and the number of elements in these input parameters is used as the batch size. The value of each element in these parameters indicates the sum of sequence lengths of the current batch and all previous batches. Therefore, the value of the next element must be greater than or equal to the value of the previous element.
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
    • In sparse mode, only sparse = 0 and attenMask is nullptr, or sparse = 3 and attenMask is not nullptr, or sparse = 4 and the input attenMask is not nullptr are supported.
    • When dimension [object Object] of [object Object] is [object Object]:
      • The layout can be [object Object] or [object Object].
      • Paged attention must be enabled. In this case, the length of [object Object] is equal to the batch size of [object Object]/[object Object], indicating the actual length of each batch. The value must be less than or equal to [object Object].
      • Dimension [object Object] for each batch of [object Object] can be set to a value from [object Object] to [object Object].
      • Dimension [object Object] for [object Object] must be set to [object Object]/[object Object]/[object Object], and dimension [object Object] for [object Object] and [object Object] must be [object Object].
      • [object Object] and [object Object] must not be empty, and dimension [object Object] for both [object Object] and [object Object] must be set to [object Object].
      • Left padding, tensor list, PSE, prefix, fake-quantization, full quantization, and post-quantization are not supported.
      • When the layout is [object Object], [object Object] cannot be enabled.
    • When dimension [object Object] of [object Object] is not [object Object]:
      • When [object Object] and [object Object] are empty, if the layout is [object Object], [object Object], [object Object], and [object Object] must be equal and less than or equal to [object Object], or [object Object] and [object Object] must be equal to [object Object] and [object Object] must be equal to [object Object]/[object Object]; if the layout is [object Object], [object Object] and [object Object] must be equal to [object Object]/[object Object], and [object Object] must be equal to [object Object]. When [object Object] and [object Object] are not null, [object Object], [object Object], and [object Object] must be equal to [object Object].
      • The layout can be [object Object] or [object Object].
      • If the layout is [object Object], the data type can only be FLOAT16 or BFLOAT16. If the layout is [object Object], the data type can only be BFLOAT16.
      • If the layout is [object Object], when the head configuration is GQA/MQA (that is, the [object Object] and [object Object] parameters must be completely passed, [object Object] must be an integer multiple of [object Object], and the two values must be different), the following constraints apply:
        • When the data type is FLOAT16 or BFLOAT16, sparse = 0 and attenMask is nullptr, or sparse = 3 and the optimized attenMask is transferred.
          • When Q_D, K_D, and V_D are equal and less than or equal to 256, or when Q_D and K_D are equal to 192 and V_D is equal to 128 or 192, sparse = 4 is supported and the optimized attenMask is input. In this case, preTokens >= –actualSeqLengths, nextTokens >= –actualSeqLengthsKv, and preTokens + nextTokens >= 0 must be met.
        • [object Object] can only be [object Object], indicating the high-precision mode without invalid row correction.
        • Paged attention is supported. The KV cache layout format supports BnBsH (blocknum, blocksize, H), where H is less than or equal to 65535, and blockSize is less than or equal to 128 and 16-byte aligned.
      • If the layout is [object Object], when the head configuration is MHA, the following constraints apply:
        • If the data type is FLOAT16 or BFLOAT16, sparse = 0 and attenMask is nullptr, or sparse = 3 or 4 and the optimized attenMask is input.
        • If the data type is FLOAT16, innerPrecise = 0 and innerPrecise = 1 are supported.
        • If the data type is BFLOAT16, only innerPrecise = 0 is supported.
        • Paged attention is supported. The KV cache layout format supports BnBsH (blocknum, blocksize, H), where H is less than or equal to 65535, and blockSize is less than or equal to 128 and 16-byte aligned.
      • If the layout is [object Object], paged attention is not supported.
      • When the sparse mode is [object Object], [object Object] must be less than [object Object] for each batch.
      • Left padding, tensor list, PSE, paged attention, prefix, fake-quantization, full quantization, and post-quantization are not supported.
      • The number of elements in [object Object] and [object Object] must be less than or equal to [object Object].
  • Ascend 950PR/Ascend 950DT:
    • TND is supported.
    • Left padding, tensor list, pseType = 0, and prefix are not supported.

• Constraints on [object Object] and [object Object] in the MLA structure:

  • The data type and format of [object Object] must be the same as those of [object Object].
  • The data type and format of [object Object] must be the same as those of [object Object].
  • Both [object Object] and [object Object] are configured, or neither of them is configured. Configuring only one of the parameters is not supported.
  • When [object Object] and [object Object] are input, only the following features are supported:
    • [object Object] can only be FP16 or BF16.
    • Dimension [object Object] in [object Object] can only be [object Object]/[object Object].
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
      • When [object Object] is greater than [object Object] (that is, in the MTP mode), the [object Object] parameter can be configured only when [object Object] is [object Object]. Other layouts are not supported.
      • When dimension [object Object] of [object Object] is [object Object]:
        • When configuring [object Object], ensure that dimension [object Object] in [object Object] is set to a value from [object Object] to [object Object] and dimension [object Object] is [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], or [object Object]. In the shape of [object Object], dimension [object Object] is [object Object], and the values of other dimensions are the same as those of [object Object].
        • When configuring [object Object], ensure that dimension [object Object] in [object Object] is [object Object] and dimension [object Object] is [object Object]. In the shape of [object Object], dimension [object Object] is [object Object], and the values of other dimensions are the same as those of [object Object].
        • sparse: When Q_S is equal to 1, only sparse = 0 and attenMask = nullptr are supported. When Q_S is greater than 1, only sparse = 3 and mask are supported.
        • The ND and NZ inputs are supported for [object Object], [object Object], and [object Object]. The input format for NZ is [object Object].
        • [object Object] can be [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], or [object Object]. In the NZ input format, [object Object] cannot be [object Object] or [object Object].
        • When paged attention must be enabled, [object Object] can be [object Object] or [object Object]. In the NZ input format, [object Object] cannot be set to [object Object].
        • Left padding, tensor list, PSE, prefix, fake-quantization, full quantization, and post-quantization are not supported.
      • When dimension [object Object] of [object Object] is [object Object]:
        • [object Object] can be [object Object] or [object Object].
        • When configuring [object Object], ensure that dimension [object Object] in the shape of [object Object] is [object Object], and the values of other dimensions are the same as those of [object Object].
        • When configuring [object Object], ensure that dimension [object Object] in the shape of [object Object] is [object Object], and the values of other dimensions are the same as those of [object Object].
        • Other restrictions are the same as those when the layout is [object Object] or [object Object].
        • Left padding, tensor list, PSE, paged attention, prefix, fake-quantization, full quantization, and post-quantization are not supported.
    • Ascend 950PR/Ascend 950DT:
      • When dimension [object Object] of [object Object] is [object Object]:
        • During queryRope configuration, the value of s for the query ranges from 1 to 16, the value of n is 1, 2, 4, 8, 16, 32, 64, or 128, and the value of d is 512. In the shape of queryRope, the values of b, n, and s are the same as those of query, and the value of d is 64.
        • During keyRope configuration, the value of n for the key is 1 and the value of d is 512. In the shape of keyRope, the values of b, n, and s are the same as those of key, and the value of d is 64.
        • sparse: sparse = 0, sparse = 3 with mask input, and sparse = 4 with mask input are supported.
        • key and value support ND input.
        • inputLayout: BSH, BSND, BNSD, TND.
        • The actualSeqLengths and actualSeqLengthsKv parameters are supported. When Q_S is greater than 1 (that is, MTP) and the normal part of key&value reuses the same data, the actualSeqLengths parameter can be configured only when the inputLayout is TND.
        • PSE is not supported.
      • When dimension [object Object] of [object Object] is [object Object]:
        • During queryRope configuration, the values of b, n, and s in the shape of queryRope must be the same as those of query, and the value of d is 64.
        • During keyRope configuration, the values of b, n, and s in the shape of keyRope must be the same as those of key, and the value of d is 64.
        • inputLayout: BSH, BSND, BNSD, BNSD_BSND, TND;
        • Paged attention, prefix, fake-quantization, full quantization, and post-quantization are not supported.
        • When kv is a tensor list, the value of b in the shape of keyRope must be the same as the length of the tensor list, the values of n and s must be the same as those of each tensor in the tensor list, and the value of d is 64.
        • PSE is not supported.

• Restrictions on [object Object]: [object Object] must be exactly divided by [object Object]. In the BSND, BNSD, BNSD_BSND, and TND scenarios, the value must be the same as the shape value of the key/value N axis in the shape. Otherwise, the execution is abnormal.

  • Ascend 950PR/Ascend 950DT:
    • In fake-quantization and full quantization scenarios, the ratio of numHeads to numKeyValueHeads cannot be greater than 64. In MLA decode scenarios, the ratio of numHeads to numKeyValueHeads cannot be greater than 128. In non-quantization and MLA prefill scenarios, the ratio of numHeads to numKeyValueHeads can be greater than 64 only when the D axis is 64 or 128. Other D axes are not supported.

• Restrictions on [object Object]:

  [object Object]

• Restrictions on [object Object]:

  • There are four modes ([object Object]0[object Object], [object Object]1[object Object], [object Object]2[object Object], and [object Object]3[object Object]) in total, represented by 2-bit combinations. Bit 0 indicates whether to use the high-precision or high-performance mode, and bit 1 indicates whether to perform invalid row correction.

    [object Object]

  • Note: The high-precision and high-performance modes are applicable to both BFLOAT16 and INT8. Invalid row correction takes effect for FLOAT16, BFLOAT16, and INT8. Currently, [object Object] and [object Object] are reserved values. If any row in the mask involved in computation consists entirely of [object Object]s, precision may degrade. In this case, you can set this parameter to [object Object]2[object Object] or [object Object]3[object Object] to enable invalid row correction to improve the precision. However, this configuration deteriorates the performance. If the operator can determine that invalid rows exist, invalid row correction is automatically enabled, such as in scenarios where [object Object]sparseMode[object Object] is [object Object]3[object Object] and [object Object]Sq[object Object] is greater than [object Object]Skv[object Object].

• Restrictions on [object Object]:

  [object Object]
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], if [object Object] is [object Object], passing a value other than [object Object], [object Object], [object Object], [object Object], [object Object], or [object Object] will result in an execution error. If [object Object] is greater than or equal to [object Object], only the values [object Object] and [object Object] are supported. Other values will result in an execution error.
  • Ascend 950PR/Ascend 950DT: If a value other than 0, 1, 2, 3, 4, 5, and 6 is passed, an exception occurs.

• Restrictions on [object Object]:

  • Except for the scenario where [object Object] is [object Object] and [object Object] is [object Object], the value must be the same as that of [object Object].
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], if [object Object] is [object Object], passing a value other than [object Object], [object Object], [object Object], [object Object], [object Object], or [object Object] will result in an execution error. If [object Object] is greater than or equal to [object Object], only the values [object Object] and [object Object] are supported. Other values will result in an execution error.
  • Ascend 950PR/Ascend 950DT: If a value other than 0, 1, 2, 3, 4, 5, and 6 is passed, an exception occurs.

• Restrictions on [object Object]:

  • In the ring attention algorithm, the product of [object Object] and [object Object] is first processed to obtain [object Object]. This max value is subtracted from the product before calculating the exponential, which is then summed to yield [object Object]softmax_sum[object Object]. Finally, the log of [object Object]softmax_sum[object Object] is added back to [object Object]softmax_max[object Object] to obtain the final result.
  • When [object Object] is [object Object], the shape must be [object Object] in general. When [object Object] is [object Object] or [object Object], the shape must be [object Object].
  • When [object Object] is [object Object], if the [object Object] tensor is not [object Object], the tensor data is returned directly. If [object Object] is [object Object], a tensor of shape [object Object] filled with zeros is returned.

• When [object Object] is greater than [object Object]

  • Restrictions on [object Object], [object Object], and [object Object]:

    • B-axis restriction

      • The B axis must be less than or equal to [object Object].
      • In non-contiguous scenarios, [object Object]batch[object Object] in the tensor list of [object Object]key[object Object] and [object Object]value[object Object] must be [object Object]1[object Object]. The number of [object Object]batch[object Object] elements is equal to [object Object]B[object Object] in [object Object]query[object Object]. [object Object]N[object Object] and [object Object]D[object Object] must be the same. Due to the tensor list restrictions, [object Object] cannot be greater than [object Object] in non-contiguous scenarios.
      • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
        • If the input type is INT8 and the D axis is not 32-byte aligned, the maximum value of the B axis is 128. If the input type is FLOAT16 or BFLOAT16 and the D-axis value is not 16-byte aligned, the B-axis value can only be [object Object]128[object Object].

    • N-axis restriction

      • Ascend 950PR/Ascend 950DT:
        • In the GQA non-quantization scenario, the value of N can be greater than 256. In the fake-quantization and full-quantization scenarios, the value of N must be less than or equal to 256.
      • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The number of N axes must be less than or equal to 256.

    • The D axis must be less than or equal to 512. If [object Object]inputLayout[object Object] is [object Object]BSH[object Object] or [object Object]BSND[object Object], it is recommended that N × D be less than [object Object]65535[object Object].

    • The S axis must be less than or equal to [object Object] (20M). In some long sequence scenarios, if the computation load is too large, the operator execution may time out (an AI Core error is reported, and [object Object] is [object Object]). In this case, S axis splitting is recommended. Note: The computation load is affected by parameters such as [object Object], [object Object], [object Object], and [object Object]. Larger values indicate larger computation loads. The following lists some typical scenarios with long sequences (that is, the product of [object Object]B[object Object], [object Object]S[object Object], [object Object]N[object Object], and [object Object]D[object Object] is large):

      [object Object]

    • D-axis restriction: [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

      • If the data type of query, key, value, or attentionOut is INT8, the D axis must be 32-pixel aligned. If the data type of query, key, value, or attentionOut is INT4, the D axis must be 64-pixel aligned. If their data types are all FLOAT16 or BFLOAT16, the D axis must be 16-pixel aligned.

    • Ascend 950PR/Ascend 950DT:

      • Non-quantization scenario: The query, key, and value types are all FLOAT16 or BFLOAT16, and the D-axis ranges from 1 to 512.
      • Full quantization scenario: The query, key, and value types are all INT8, and the D-axis ranges from 1 to 512.
      • Fake-quantization scenario: The query type is FLOAT16 or BFLOAT16, and the key and value types are INT8/HIFLOAT8/FLOAT8_E4M3FN/FLOAT4_E2M1/INT4(INT32). When the key and value types are FLOAT4_E2M1/INT4(INT32), the D axis of the query and the D axis of the key and value must be 64-byte aligned. (When the key and value types are INT32, the D axis of the key and value must be 8-byte aligned.)

  • For the input parameter [object Object], the value must be a non-negative number.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], the valid sequence length of each batch in this input parameter must be less than or equal to the sequence length of the corresponding batch in [object Object]. If the input length of [object Object] is [object Object], all batches use the same [object Object]. If the input length is greater than or equal to the batch size, the first N elements (where N equals the batch size) of [object Object] are used. Other lengths are not supported. When the inputLayout of the query is TND/NTD_TND,
    • Ascend 950PR/Ascend 950DT: The meaning and interception condition of this input parameter vary according to the inputLayout. If the inputLayout is not TND, this input parameter is optional. The length of this input parameter is 1 or greater than or equal to the batch size of the query. The value in this input parameter indicates the actual length of each batch, which must be less than or equal to Q_S. If the inputLayout is TND, this input parameter is mandatory. The bth value indicates the accumulated length of the S axis of the first b batches. The values are arranged in ascending order (greater than or equal to the previous value). The length of this input parameter indicates the total number of batches.

  • For the input parameter [object Object], the value must be a non-negative number.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], the valid sequence length of each batch in this input parameter must be less than or equal to the sequence length of the corresponding batch in [object Object] or [object Object]. If the input length of [object Object] is [object Object], all batches use the same [object Object]. If the input length is greater than or equal to the batch size, the first N elements (where N equals the batch size) of [object Object] are used. Other lengths are not supported. Applicable to the scenario where the inputLayout of the key/value is TND/NTD_TND.
    • Ascend 950PR/Ascend 950DT: The meaning and interception conditions of this parameter vary according to the inputLayout. When inputLayout is not TND, this parameter is optional. The length of this parameter is 1 or greater than or equal to the batch value of key/value. The value of this parameter indicates the actual length of each batch, and the value must be less than or equal to KV_S. When inputLayout is TND, this parameter is mandatory. In the non-PA scenario, the bth value indicates the accumulated length of the S axis of the first b batches. The values are arranged in ascending order (greater than or equal to the previous value). The length of this parameter indicates the total number of batches. In the PA scenario, the length of this parameter is equal to the batch value of key/value, indicating the actual length of each batch. The value must be less than or equal to KV_S.

  • Currently, [object Object] can only be set to [object Object], [object Object], [object Object], [object Object], or [object Object]. An error will be reported if it is set to other values.

    • When [object Object] is set to [object Object], if [object Object] is a null pointer or is passed in the left padding scenario, the input parameters [object Object] and [object Object] are ignored.
    • When sparseMode is set to 2, 3, or 4, the shape of attenMask must be S,S, 1,S,S, or 1,1,S,S. The value of S must be fixed to 2048. In addition, you need to ensure that the input attenMask is a lower triangle. If attenMask is nullptr or the input shape is incorrect, an error is reported.
    • When [object Object] is set to [object Object], [object Object], or [object Object], the input parameters [object Object] and [object Object] are ignored, and their values are assigned based on related rules.

  • In the synthesis parameter scenario of KV cache dequantization, only when [object Object] is of the FLOAT16 type can [object Object] and [object Object] of the INT8 type be dequantized to FLOAT16. If the product of the data ranges of the input parameters [object Object]key[object Object] and [object Object]value[object Object] and the data range of the input parameter [object Object]antiquantScale[object Object] must be within the range of (–1, 1), the high-performance mode can ensure precision. Otherwise, the high-precision mode needs to be enabled to ensure precision.

  • Paged attention scenario:

    • The prerequisite for enabling paged attention is that [object Object] exists and is valid, and [object Object] and [object Object] are arranged in a continuous memory based on the indexes in [object Object]. In this scenario, [object Object] of [object Object] and [object Object] is invalid. [object Object] is filled with block IDs. Currently, the validity of block IDs is not verified. You need to ensure the validity of block IDs.
    • [object Object] is a user-defined parameter. Its value affects the paged attention performance. When paged attention is enabled, the value of [object Object] must be a multiple of [object Object], ranging from [object Object] to [object Object]. Generally, paged attention can improve the throughput but deteriorate the performance.
    • In the paged attention scenario, if the input KV cache layout is BnBsH [object Object] and the product of [object Object] multiplied by [object Object] exceeds [object Object], an error will be reported due to hardware instruction constraints. This problem can be solved by enabling GQA (decreasing [object Object]KV_N[object Object]) or adjusting the KV cache layout to BnNBsD [object Object](BlockNum, KV_N, BlockSize, D)[object Object]. When [object Object] of [object Object] is [object Object] or [object Object], the KV cache layout can be BnBsH or BnNBsD. When [object Object] of [object Object] is [object Object] or [object Object], the KV cache layout can only be BnBsH. The value of [object Object]BlockNum[object Object] cannot be less than the sum of blocks in each batch calculated based on [object Object]actualSeqLengthsKv[object Object] and [object Object]BlockSize[object Object]. The shapes of [object Object]key[object Object] and [object Object]value[object Object] must be the same.
    • Paged attention fake-quantization scenario
      • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], the data type of [object Object] can be FLOAT16 or BFLOAT16, and the data types of [object Object] and [object Object] can be INT8.
      • Ascend 950PR/Ascend 950DT: The key and value can be of type FLOAT16/BFLOAT16/INT8/HIFLOAT8/FLOAT8_E4M3FN/FLOAT4_E2M1/INT4(INT32).
    • Paged attention full-quantization scenario
      • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], the data type of [object Object] cannot be INT8.
      • Ascend 950PR/Ascend 950DT: The query dtype can be INT8.
    • Paged attention does not support the tensor list or left padding.
    • In the paged attention scenario, [object Object] must be passed.
    • In the paged attention scenario, [object Object] must be two-dimensional. The length of the first dimension must be equal to [object Object], and the length of the second dimension must be greater than or equal to [object Object] (the maximum number of blocks corresponding to [object Object] in different batches).
    • When paged attention is enabled, the input [object Object] must be greater than or equal to [object Object] × [object Object] in the following scenarios:
      • [object Object] is passed, for example, when the mask shape is [object Object].
      • [object Object] is passed, for example, when the [object Object] shape is [object Object].

  • Left padding for [object Object]:

    • The transfer start point of [object Object] is calculated as follows: [object Object] – [object Object] – [object Object]. The transfer end point of [object Object]query[object Object] is calculated as follows: [object Object]Q_S[object Object] – [object Object]queryPaddingSize[object Object]. The transfer start point of [object Object]query[object Object] cannot be less than [object Object]0[object Object], while the end point cannot be greater than [object Object]Q_S[object Object]. Otherwise, the result will not meet the expectation.
    • If [object Object] is less than [object Object], it will be set to [object Object].
    • It must be enabled together with [object Object]. Otherwise, the default scenario is right padding for [object Object].
    • It does not support paged attention and cannot be enabled together with [object Object].
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The scenario where Q is BF16/FP16 and KV is INT4 is not supported.

  • Left padding for [object Object]:

    • The transfer start point of [object Object] and [object Object] is calculated as follows: [object Object] – [object Object] – [object Object]. The transfer end point of [object Object]key[object Object] and [object Object]value[object Object] is calculated as follows: [object Object]KV_S[object Object] – [object Object]kvPaddingSize[object Object]. The transfer start point of [object Object]key[object Object] and [object Object]value[object Object] cannot be less than [object Object]0[object Object], while the end point cannot be greater than [object Object]KV_S[object Object]. Otherwise, the result will not meet the expectation.
    • If [object Object] is less than [object Object], it will be set to [object Object].
    • It must be enabled together with [object Object]. Otherwise, the default scenario is right padding for [object Object].
    • It does not support paged attention and cannot be enabled together with [object Object].
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The scenario where Q is BF16/FP16 and KV is INT4 is not supported.

  • When the output is of type INT8 and [object Object] and [object Object] are per-channel, left padding, ring attention, or non-32-byte alignment of the D axis is not supported.

  • When the output is of type INT8, [object Object] cannot be [object Object] and [object Object] or [object Object] cannot be negative.

  • When the output is of type INT8, if the input parameter[object Object] is a non-null pointer and a non-null tensor, and [object Object], [object Object], and [object Object] meet the following conditions, certain rows of the matrix will not be involved in computation, resulting in a computation result error. In this scenario, the computation will be intercepted. (Solution: To prevent interception, perform post-quantization outside the FIA interface.)

    • When [object Object] is [object Object] and [object Object] is a non-null pointer, interception occurs if for any batch: [object Object] – [object Object] – [object Object] – [object Object] > [object Object], or [object Object] < [object Object].
    • When [object Object] is [object Object] or [object Object], interception does not occur.
    • When [object Object] is [object Object], interception occurs if for any batch: [object Object] + [object Object] – [object Object] < [object Object].
    • When [object Object] is [object Object], interception occurs if for any batch: [object Object] < [object Object], or [object Object] + [object Object] + [object Object] – [object Object] < [object Object].

  • Restrictions on [object Object]:

    • This function is supported when the data type of [object Object] is FLOAT16, BFLOAT16, or INT8.
    • When the data type of [object Object] is FLOAT16 and [object Object] exists, the high-precision mode is forcibly used. The corresponding restrictions are the same as those of the high-precision mode.
    • [object Object] must be greater than or equal to [object Object] of [object Object], and [object Object] must be greater than or equal to [object Object] of [object Object]. In the prefix scenario, the value of KV_S must be greater than or equal to the sum of the value of actualSharedPrefixLenand the S length of key.
    • Ascend 950PR/Ascend 950DT: In non-quantization and full-quantization scenarios, there is no alignment restriction.

  • Restrictions on prefix parameters:

    • Both [object Object] and [object Object] must be either null or non-null.
    • If neither [object Object] nor [object Object] is null, the dimensions and data types of [object Object], [object Object], [object Object], and [object Object] must be the same.
    • If neither [object Object] nor [object Object] is null, the first dimension (batch) of the shape of [object Object] must be [object Object]. When the layout is [object Object] or [object Object], the N and D axes must be the same as those of [object Object]. When the layout is [object Object], the H axis must be the same as that of [object Object]. The same rules apply to [object Object]. [object Object] of [object Object] and [object Object] must be the same.
    • When [object Object] exists, its shape must be [object Object], and its value cannot be greater than [object Object] of [object Object] and [object Object].
    • The sum of [object Object] of the public prefix and [object Object] of [object Object] or [object Object] must meet the original restriction on [object Object] of [object Object] or [object Object].
    • The prefix does not support paged attention, left padding, or tensor list.
    • In the prefix scenario, when [object Object] is [object Object] or [object Object] and [object Object] is passed, [object Object] must be greater than or equal to the sum of [object Object] and [object Object] of [object Object].
    • In the prefix scenario, the input [object Object] cannot be all INT8.

  • KV fake-quantization parameter separation:

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
      • [object Object] and [object Object] must be the same.
      • Both [object Object] and [object Object] must be either null or non-null. Both [object Object] and [object Object] must be either null or non-null.
      • If both keyAntiquantScale and valueAntiquantScale are not empty, their shapes must be the same. If both keyAntiquantOffset and valueAntiquantOffset are not empty, their shapes must be the same.
      • Only the per-token and per-channel modes are supported. In per-token mode, the shapes of the two parameters must be both [object Object], and the data type is fixed at FLOAT32. In per-channel mode, the shapes of the two parameters must be [object Object], [object Object], or [object Object], and the data type is fixed at BF16.
      • When both fake-quantization parameters and KV separation quantization parameters are passed, the KV separation quantization parameters take effect.
      • When [object Object] and [object Object] are non-null, [object Object] of [object Object] must be less than or equal to [object Object].
      • When [object Object] and [object Object] are non-null, the data type of [object Object] must be BFLOAT16, the data types of [object Object] and [object Object] must be INT8, and the data type of the output must be BFLOAT16.
      • When [object Object] and [object Object] are non-null, the tensor list, left padding, and paged attention functions are not supported.
    • Ascend 950PR/Ascend 950DT:
      • Except when [object Object] is [object Object] and [object Object] is [object Object], the values of [object Object] and [object Object] must be the same.
      • Both [object Object] and [object Object] must be either null or non-null. Both [object Object] and [object Object] must be either null or non-null.
      • If both keyAntiquantScale and valueAntiquantScale are not empty, their shapes must be the same, except when keyAntiquantMode is 0 and valueAntiquantMode is 1. If both keyAntiquantOffset and valueAntiquantOffset are not empty, their shapes must be the same, except when keyAntiquantMode is 0 and valueAntiquantMode is 1.
      • The following nine modes are supported: per-channel, per-tensor, per-token, per-tensor + per-head, per-token + per-head, per-token + paged attention to manage scale/offset, per-token + per-head + paged attention to manage scale/offset, per-channel for [object Object] + per-token for [object Object], and per-channel for [object Object] + per-token-group for [object Object]. In the following description, [object Object] indicates [object Object].
      • Per-channel mode: The shapes of the two parameters can be (1, N, 1, D), (1, N, D), (1, H), (N, 1, D), (N, D), or (H). The parameter data type is the same as the query data type. This mode is supported when the key and value data types are INT8, INT4 (INT32), HIFLOAT8, or FLOAT8_E4M3FN. When the key and value data types are HIFLOAT8 or FLOAT8_E4M3FN, antiquantOffset is not supported.
      • Per-tensor mode: The shapes of the two parameters are both (1), and the data type is the same as the query data type. This mode is supported when the key and value data types are INT8.
      • Per-token mode: The shapes of the two parameters can be (1, B, S) or (B, S), and the data type is fixed to FLOAT32. This mode is supported when the key and value data types are INT8 or INT4 (INT32).
      • Per-tensor + per-head mode: The shapes of the two parameters are both [object Object], their data type is the same as that of [object Object], and the data types of [object Object] and [object Object] are INT8.
      • Per-channel for keys and per-token for values: For keys, the per-channel mode is supported, and the shapes of the two parameters can be (1, N, 1, D), (1, N, D), (1, H), (N, 1, D), (N, D), or (H), and the parameter data type is the same as the query data type. For values, the per-token mode is supported, and the shapes of the two parameters are both (1, B, S), and the data type is fixed to FLOAT32. This mode is supported when the key and value data types are INT8 or INT4 (INT32). When the key and value data types are INT8, the query and output support only FLOAT16.
      • In per-token-group mode, the shape of antiquantScale is (1, B, N, S, D/32), the data type is fixed to FLOAT8_E8M0, and the antiquantOffset is not supported. This mode is supported when the data type of key and value is FLOAT4_E2M1.
      • Per-token + per-head mode: The shapes of the two parameters are both [object Object], their data type is fixed at FLOAT32, and the data types of [object Object] and [object Object] are INT8 or INT4 (INT32).
      • Per-token + paged attention to manage scale/offset: The shapes of the two parameters are both [object Object], their data type is fixed at FLOAT32, and the data types of [object Object] and [object Object] are INT8.
      • Per-token + per-head + paged attention to manage scale/offset: The shapes of the two parameters are both [object Object], their data type is fixed at FLOAT32, and the data types of [object Object] and [object Object] are INT8.
      • When both fake-quantization parameters and KV separation quantization parameters are passed, the KV separation quantization parameters take effect.
      • In the INT4 (INT32) fake-quantization scenario where only KV fake-quantization parameter separation is supported, the following modes are supported:
        • Per-channel
        • Per-token
        • Per-token + per-head
        • Per-channel for [object Object] + per-token for [object Object]
      • Post-quantization is not supported in some fake-quantization scenarios.
        • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Post-quantization is not supported in the INT4 (INT32) fake-quantization scenario.
        • Ascend 950PR/Ascend 950DT: Post-quantization is not supported in the INT4 (INT32) and FLOAT4_E2M1 fake-quantization scenarios.

• When [object Object] is equal to [object Object]:

  • Constraints on [object Object], [object Object], and [object Object]:
    • The B axis must be less than or equal to 65536, and the D axis must be less than or equal to 512.
    • N-axis restriction
      • Ascend 950PR/Ascend 950DT:
        • In the GQA non-quantization scenario, the N axis can be greater than 256. In the fake-quantization and full-quantization scenarios, the N axis must be less than or equal to 256.
      • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The number of N axes must be less than or equal to 256.
    • The input types of [object Object], [object Object], and [object Object] cannot be all INT8.
    • In INT4 (INT32) fake-quantization scenarios, the aclnn single-operator call supports KV inputs in either INT4 format or INT4-packed INT32 format. Using [object Object] to generate INT4 data is recommended, which stores eight INT4 elements within one INT32.
    • In INT4 (INT32) fake-quantization scenarios, if KV INT4 values are packed into INT32 inputs, the [object Object], [object Object], or [object Object] dimensions of KV must be 1/8 of their actual values (the same applies to prefix).
    • Restrictions on the D axis for [object Object] and [object Object] in specific data types
      • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], when the input type of [object Object] and [object Object] is INT4 (INT32), the D axis must be 64-byte aligned (or 8-byte aligned for INT32).
      • Ascend 950PR/Ascend 950DT: When the input types of key and value are FLOAT4_E2M1/INT4(INT32), the D axis of query and the D axis of key and value must be 64-aligned. (INT32 supports only 8-aligned D axis for key and value.)
  • For the input parameter [object Object], the value must be a non-negative number.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], when [object Object] of [object Object] is not [object Object], this parameter is invalid if [object Object] is [object Object]. When the inputLayout of the query is TND/TND_NTD:
    • Ascend 950PR/Ascend 950DT: The meaning and interception conditions of this parameter vary according to the inputLayout. When the inputLayout is not TND, this parameter is optional. The length of this parameter is 1 or greater than or equal to the batch value of query. The value of this parameter indicates the actual length of each batch, and the value must be less than or equal to Q_S. When the inputLayout is TND, this parameter is mandatory. The bth value indicates the accumulated length of the S axis of the first b batches. The values must be in ascending order (greater than or equal to the previous value), and the length of this parameter indicates the total number of batches.
  • For the input parameter [object Object], the value must be a non-negative number.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], the valid sequence length of each batch in this input parameter must be less than or equal to the sequence length of the corresponding batch in [object Object] or [object Object]. If the input length of [object Object] is [object Object], all batches use the same [object Object]. If the input length is greater than or equal to the batch size, the first N elements (where N equals the batch size) of [object Object] are used. Other lengths are not supported. When the inputLayout of the key/value is TND/TND_NTD:
    • Ascend 950PR/Ascend 950DT: The meaning and interception conditions of this input parameter vary according to the inputLayout. When the inputLayout is not TND, this input parameter is optional. The length of this input parameter is 1 or greater than or equal to the batch size of the key/value. The value of this input parameter indicates the actual length of each batch, and the value must be less than or equal to KV_S. When the inputLayout is TND, this input parameter is mandatory. In the non-PA scenario, the bth value indicates the accumulated length of the S axis of the first b batches. The values are arranged in ascending order (greater than or equal to the previous value). The length of this input parameter indicates the total number of batches. In the PA scenario, the length of this input parameter is equal to the batch size of the key/value, indicating the actual length of each batch. The value must be less than or equal to KV_S.
  • Paged attention scenarios:
    • The prerequisite for enabling paged attention is that [object Object] exists and is valid, and [object Object] and [object Object] are arranged in a continuous memory based on the indexes in [object Object]. In this scenario, [object Object] of [object Object] and [object Object] is invalid.
      • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], the data types of [object Object] and [object Object] can be FLOAT16, BFLOAT16, or INT8.
      • Ascend 950PR/Ascend 950DT: The key and value data types can be FLOAT16/BFLOAT16/INT8/HIFLOAT8/FLOAT8_E4M3FN/FLOAT4_E2M1/INT4(INT32).
    • [object Object] is a user-defined parameter. Its value affects the paged attention performance. When paged attention is enabled, a non-zero value must be passed for [object Object], and the maximum value is [object Object]. Generally, paged attention improves throughput but reduces performance.
      • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], if the input type of [object Object] and [object Object] is FLOAT16 or BFLOAT16, 16-byte alignment is required. If the input type of [object Object] and [object Object] is INT8, 32-byte alignment is required (128-byte alignment is recommended).
      • Ascend 950PR/Ascend 950DT: When the key and value inputs are of type FLOAT16 or BFLOAT16, the alignment must be 16. When the key and value inputs are of type INT8/HIFLOAT8/FLOAT8_E4M3FN, the alignment must be 32. When the key and value inputs are of type FLOAT4_E2M1/INT4(INT32), the alignment must be 64.
    • In the paged attention scenario, when [object Object] of [object Object] is [object Object] or [object Object], the KV cache layout can be BnBsH [object Object] or BnNBsD [object Object]. When [object Object] of [object Object] is [object Object] or [object Object], the KV cache layout can only be BnBsH. The value of [object Object]BlockNum[object Object] cannot be less than the sum of blocks in each batch calculated based on [object Object]actualSeqLengthsKv[object Object] and [object Object]BlockSize[object Object]. The shapes of [object Object]key[object Object] and [object Object]value[object Object] must be the same.
    • In the paged attention scenario, the performance is generally better when the KV cache layout is BnNBsD than when it is BnBsH. Therefore, BnNBsD is recommended.
    • In the paged attention scenario, if the input KV cache layout is BnBsH and [object Object] × [object Object] exceeds 64 KB, an error will be reported due to hardware instruction constraints. This problem can be solved by enabling GQA (decreasing [object Object]) or adjusting the KV cache layout to BnNBsD.
    • Paged attention does not support the tensor list or left padding:
      • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The scenario where Q is BF16/FP16 and KV is INT4 (INT32) is not supported.
      • Ascend 950PR/Ascend 950DT: The scenario where Q is BF16/FP16 and KV is INT4 (INT32) is supported.
    • In the paged attention scenario, [object Object] must be passed.
    • In the paged attention scenario, [object Object] must be two-dimensional. The length of the first dimension must be equal to [object Object], and the length of the second dimension must be greater than or equal to [object Object] (the maximum number of blocks corresponding to [object Object] in each batch).
    • When paged attention is enabled, the input [object Object] must be greater than or equal to [object Object] second dimension × [object Object] in the following scenarios:
      • [object Object] is enabled, for example, when the mask shape is [object Object].
      • [object Object] is enabled, for example, when the [object Object] shape is [object Object].
      • When the fake-quantization per-token mode is enabled, the shape of the input parameters [object Object] and [object Object] is both [object Object].
      • Per-token + per-head mode: The shapes of the two parameters are both [object Object], their data type is fixed at FLOAT32, and the data types of [object Object] and [object Object] are INT8 or INT4 (INT32).
      • Per-token-group mode: The shape of [object Object] is [object Object], and its data type is fixed at FLOAT8_E8M0. [object Object] is not supported. The data types of [object Object] and [object Object] are FLOAT4_E2M1.
  • Left padding for [object Object] and [object Object]:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], scenarios where [object Object] is of type BF16/FP16 and [object Object] is of type INT4 (INT32) are not supported.
    • Ascend 950PR/Ascend 950DT: The scenario where Q is BF16/FP16 and KV is INT4 (INT32) is supported, and there is no restriction on the QKV data type.
    • The start point of the kvCache movement is calculated as follows: KV_S – kvPaddingSize – actualSeqLengthsKv. The transfer end point of [object Object] is calculated as follows: [object Object] – [object Object]. If the transfer start point or end point is less than [object Object], the returned data is all 0s.
    • If [object Object] is less than [object Object], it will be set to [object Object].
    • It must be enabled together with [object Object]. Otherwise, the default scenario is right padding for [object Object].
    • Paged attention and tensor list are not supported. Otherwise, the default scenario is right padding for [object Object].
    • When it is enabled together with [object Object], ensure that the meaning of [object Object] is correct, that is, invalid data can be correctly masked. Otherwise, precision issues may occur.
  • Restrictions on [object Object]:
    • The data types of [object Object] and [object Object] must be the same.
  • Constraints on KV fake-quantization parameter separation:
    • Except when [object Object] is [object Object] and [object Object] is [object Object], the values of [object Object] and [object Object] must be the same.
    • Both [object Object] and [object Object] must be either null or non-null. Both [object Object] and [object Object] must be either null or non-null.
    • If neither [object Object] nor [object Object] is null, their shapes must be the same, except when [object Object] is [object Object] and [object Object] is [object Object]. If neither [object Object] nor [object Object] is null, their shapes must be the same, except when [object Object] is [object Object] and [object Object] is [object Object].
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The following eight modes are supported: per-channel, per-tensor, per-token, per-tensor + per-head, per-token + per-head, per-token + paged attention to manage scale/offset, per-token + per-head + paged attention to manage scale/offset, and per-channel for [object Object] + per-token for [object Object]. In the following description, [object Object] indicates [object Object].
      • Per-channel mode: The shapes of the two parameters can be [object Object], [object Object], or [object Object], their data type is the same as that of [object Object], and the data types of [object Object] and [object Object] are INT8 or INT4 (INT32).
      • Per-tensor mode: The shapes of the two parameters are both [object Object], their data type is the same as that of [object Object], and the data types of [object Object] and [object Object] are INT8.
      • Per-token mode: The shapes of the two parameters are both [object Object], their data type is fixed at FLOAT32, and the data types of [object Object] and [object Object] are INT8 or INT4 (INT32).
      • Per-tensor + per-head mode: The shapes of the two parameters are both [object Object], their data type is the same as that of [object Object], and the data types of [object Object] and [object Object] are INT8.
      • Per-channel for [object Object] + per-token for [object Object] mode: In per-channel for [object Object], the shapes of the two parameters can be [object Object], [object Object], or [object Object], and their data type is the same as that of [object Object]. In per-token for [object Object], the shapes of the two parameters are both [object Object], and their data type is fixed at FLOAT32. The data types of [object Object] and [object Object] are INT8 or INT4 (INT32). When the data types of [object Object]key[object Object] and [object Object]value[object Object] are INT8, only the data types of [object Object]query[object Object] and [object Object]attentionOut[object Object] can be FLOAT16.
    • Ascend 950PR/Ascend 950DT: The following modes are supported: per-channel, per-tensor, per-token, per-tensor + per-head, per-token + per-head, per-token + paged attention to manage scale/offset, per-token + per-head + paged attention to manage scale/offset, per-channel for [object Object] + per-token for [object Object], and [object Object]. In the following description, [object Object] indicates [object Object].
      • Per-channel mode: The shapes of the two parameters can be (1, N, 1, D), (1, N, D), or (1, H). The data type of the parameter is the same as that of the query. This mode is supported when the data type of the key and value is INT8, INT4 (INT32), HIFLOAT8, or FLOAT8_E4M3FN. When the data type of the key and value is HIFLOAT8 or FLOAT8_E4M3FN, antiquantOffset is not supported.
      • Per-tensor mode: The shapes of the two parameters are both (1), and the data type is the same as that of the query. This mode is supported when the data type of the key and value is INT8 or INT4 (INT32).
      • Per-token mode: The shapes of the two parameters can be (1, B, S) or (B, S), and the data type is fixed to FLOAT32. This mode is supported when the data type of the key and value is INT8 or INT4 (INT32).
      • Per-tensor + per-head mode: The shapes of the two parameters are both [object Object], their data type is the same as that of [object Object], and the data types of [object Object] and [object Object] are INT8.
      • Per-channel for key and per-token for value: The key supports per-channel, and the shapes of the two parameters can be (1, N, 1, D), (1, N, D), or (1, H), and the data type of the parameter is the same as that of the query. The value supports per-token, and the shapes of the two parameters are both (1, B, S), and the data type is fixed to FLOAT32. This mode is supported when the data type of the key and value is INT8 or INT4 (INT32). When the key and value data types are INT8, the query and output support only FLOAT16.
      • In per-token-group mode, the shape of antiquantScale is (1, B, N, S, D/32), the data type is fixed to FLOAT8_E8M0, and the antiquantOffset is not supported. This is supported when the key and value data types are FLOAT4_E2M1.
      • Per-token + per-head mode: The shapes of the two parameters are both [object Object], their data type is fixed at FLOAT32, and the data types of [object Object] and [object Object] are INT8 or INT4 (INT32).
      • Per-token + paged attention to manage scale/offset: The shapes of the two parameters are both [object Object], their data type is fixed at FLOAT32, and the data types of [object Object] and [object Object] are INT8.
      • Per-token + per-head + paged attention to manage scale/offset: The shapes of the two parameters are both [object Object], their data type is fixed at FLOAT32, and the data types of [object Object] and [object Object] are INT8.
      • When both fake-quantization parameters and KV separation quantization parameters are passed, the KV separation quantization parameters take effect.
    • In the INT4 (INT32) fake-quantization scenario, only the KV fake-quantization parameters can be separated. The details are as follows:
      • Per-tensor mode
      • Per-channel
      • Per-token
      • Per-tensor+per-head mode
      • Per-token + per-head
      • Per-channel for [object Object] + per-token for [object Object]
    • Post-quantization is not supported in some fake-quantization scenarios.
      • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Post-quantization is not supported in the INT4 (INT32) fake-quantization scenario.
      • Ascend 950PR/Ascend 950DT: Post-quantization is not supported in the INT4 (INT32) and FLOAT4_E2M1 fake-quantization scenarios.
  • Constraints on prefix parameters:
    • Both [object Object] and [object Object] must be either null or non-null.
    • If both keySharedPrefixand valueSharedPrefixare not empty, the dimensions and dtypes of keySharedPrefix, valueSharedPrefix, keyand valueare the same.
    • If both keySharedPrefixand valueSharedPrefixare not empty, the first dimension batch of the shape of keySharedPrefixmust be 1. If the layout is BNSD or BSND, the N and D axes must be the same as that of key. If the layout is BSH, the H axis must be the same as that of key. The same applies to valueSharedPrefix. The values of S in keySharedPrefixand valueSharedPrefixmust be the same.
    • When [object Object] exists, its shape must be [object Object], and its value cannot be greater than [object Object] of [object Object] and [object Object].
    • The sum of [object Object] of the public prefix and [object Object] of [object Object] or [object Object] must meet the original restriction on [object Object] of [object Object] or [object Object].

[object Object]

The following example is for reference only. For details, see .