[object Object][object Object][object Object]undefined[object Object]

• API function: Fuses the query and key operators into one to improve the performance of the inference network. Computes the rotary positional encoding and updates the computation result in place.
• Formula:$$query\_q1 = query[..., : query.shape[-1] // 2]$$ $$query\_q2 = query[..., query.shape[-1] // 2 :]$$ $$query\_rotate = torch.cat((-query\_q2, query\_q1), dim=-1)$$ $$key\_k1 = key[..., : key.shape[-1] // 2]$$ $$key\_k2 = key[..., key.shape[-1] // 2 :]$$ $$key\_rotate = torch.cat((-key\_k2, key\_k1), dim=-1)$$ $$q\_embed = (query * cos) + query\_rotate * sin$$ $$k\_embed = (key * cos) + key\_rotate * sin$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]
  • [object Object]Atlas inference products[object Object]: The BFLOAT16 data type is not supported.

• Returns

  aclnnStatus: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  aclnnStatus: status code. For details, see .

[object Object]

• Deterministic computing:

  • [object Object] defaults to a deterministic implementation.

  • When [object Object] is 1, the first two dimensions (B and S) of the input shapes of [object Object], [object Object], [object Object], and [object Object] must be the same. When layout is 4, the first dimension (T) must be the same.

  • The last dimensions (D) of the input shapes of [object Object], [object Object], [object Object], and [object Object] must be the same.

  • The dtype of input tensors [object Object], [object Object], [object Object], and [object Object] must be the same.

  • When [object Object] is 1, the shape of [object Object] is represented by (q_b, q_s, q_n, q_d), the shape of [object Object] is represented by (q_b, q_s, k_n, q_d), and the shape of [object Object] and [object Object] is represented by (q_b, q_s, 1, q_d). b indicates batch_size, s indicates seq_length, n indicates head_num, and d indicates head_dim. When [object Object] is 4, the shape of [object Object] is represented by (q_t, q_n, q_d), the shape of [object Object] is represented by (q_t, k_n, q_d), and the shape of [object Object] and [object Object] is represented by (q_t, 1, q_d). t indicates the combined axis of b and s, n indicates head_num, and d indicates head_dim.

    • When the input is BFLOAT16, cast is 1, castSize is 4, and DtypeSize is 2.
    • When the input is FLOAT16 or FLOAT32, cast is 0, and castSize = DtypeSize (2 for FLOAT16 and 4 for FLOAT32).

    lastDim indicates the value of head_dim in the last dimension of the input shape. The UB space size to be used is calculated as follows: [object Object], If the value of [object Object] exceeds the total UB space of the current AI processor, this fusion operator cannot be used.

[object Object]

The following example is for reference only. For details, see .