[object Object][object Object][object Object]undefined[object Object]

• API function: Computes the attention backpropagation output in training scenarios, which is the backpropagation of . Compared with the API, this API has the [object Object], [object Object], [object Object], and [object Object] parameters.

• Formula:

  The forward propagation formula for attention is as follows:

  $$Y=Dropout(Softmax(Mask(\frac{QK^T}{\sqrt{d}}+pse),atten\_mask),keep\_prob)V$$

  where:

  $$Q=[query, queryRope], K=[key, keyRope]$$

  For convenience, the formula can be represented using variables $S$ and $P$:

  $$S=Mask(\frac{QK^T}{\sqrt{d}}+pse),atten\_mask$$ $$P=Dropout(Softmax(S),keep\_prob)$$ $$Y=PV$$

  Then the backward propagation formula for attention is as follows:

  $$dV=P^TdY$$ $$dQ=\frac{((dS)*K)}{\sqrt{d}}$$ $$dqRope=\frac{((dS)*kRope)}{\sqrt{d}}$$ $$dK=\frac{((dS)^T*Q)}{\sqrt{d}}$$ $$dkRope=\frac{((dS)^T*qRope)}{\sqrt{d}}$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing:
  • [object Object] defaults to a non-deterministic implementation. You can call [object Object] to enable deterministic computing.
• When this API is used together with PyTorch, ensure that the CANN package versions match the PyTorch package versions.
• B (batch size) of the input [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] must be the same.
• [object Object] of the input [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] must be TND.
• If the D values of [object Object], [object Object], and [object Object] are the same, the shapes of query and dy must be the same.
• The values of D of [object Object] and [object Object] must be the same and must be an integer multiple of 8.
• The values of D of [object Object] and [object Object] must be the same, must be an integer multiple of 8, and must be less than or equal to the value of D of [object Object] and [object Object].
• N of the input [object Object] or [object Object] can be different from N of the [object Object] or [object Object], but they must be proportional. That is, Nq/Nkv must be a non-zero integer. The value of Nq ranges from 1 to 256.
• The following uses the [object Object] TND as an example to describe the constraints on the shape:
  • T: The value ranges from 1 to 1M.
  • N: The value ranges from 1 to 256.
  • D: The value ranges from 1 to 768.
  • KeepProb: The value can be 1.
• The data format of [object Object], [object Object], and [object Object] can only be TND. T indicates the data closely arranged on the B and S axes (SeqLenQ and SeqLenKV of each batch). B (Batch) indicates the batch size of the input sample, and S (Seq-Length) indicates the length of the input sample sequence. H (Head-Size) indicates the size of the hidden layer, N (Head-Num) indicates the number of heads. D (Head-Dim) indicates the minimum unit size of the hidden layer (D = H/N).
• The constraints for [object Object] are as follows:
  • If the shape values of all [object Object] are the same and less than 2048, you are advised to use the default mode to reduce memory usage.
  • When this parameter is set to 1, 2, or 3, the configured preTokens and nextTokens do not take effect.
  • When the value is set to 0 or 4, ensure that the ranges of [object Object], [object Object], and [object Object] are consistent.
  • If no specific value is required, you are advised to set it to 0.
  • For details about the sparse modes, see .
  • When this parameter is set to 7, the optional input pseShiftOptional is not supported.
  • When this parameter is set to 8, the optional input pseShiftOptional is supported when the q and kv of each sequence are of the same length. The global PSE is generated. The q direction can be used for external splitting. The q and kv of each sequence must have the same length before external splitting. Then, actualSeqQLenOptional[0] - actualSeqKvLenOptional[0] + qStartIdxOptional - kvStartIdxOptional == 0 (experimental function).
• In some scenarios, if the computation load is too large, the operator execution may time out (AI Core error, errorStr: timeout or trap error). In this case, you are advised to perform axis splitting. Note: The computation load is affected by parameters such as B, S, N, and D. Larger values indicate larger computation loads.
• The [object Object] input supports the S length of 0 in a batch. In this case, the [object Object] input is not supported. The length of [object Object] ranges from 1 to 2K. When [object Object] is present, its maximum length is 1K.
• Constraints on the [object Object] and [object Object] parameters: The input format is fixed at[T, N, 8]. Note: T = B x S.
• The value of [object Object] must be the same as the value of N in [object Object].
• pseType can only be 1.
• [object Object] must be empty.
• [object Object] must be empty.
• [object Object] cannot be empty.

[object Object]

The following example is for reference only. For details, see .