[object Object][object Object][object Object]undefined[object Object]

• API function: Performs an optional dequantization on the input tensor (x), then splits it along the last axis into q, k, and vOut according to the specified sizeSplits. It applies Rotary Positional Embedding (RoPE) to q and k to generate qOut and kOut. Finally, kOut and vOut are quantized and updated into kCacheRef and vCacheRef based on the provided indices.

• Formula:

  $$dequantX = Dequant(x,weightScaleOptional,activationScaleOptional,biasOptional)$$ $$q,k,vOut = SplitTensor(dequantX,dim=-1,`sizeSplits`)$$ $$qOut,kOut = ApplyRotaryPosEmb(q,k,cos,sin)$$ $$quantK = Quant(kOut,scaleK,offsetKOptional)$$ $$quantV = Quant(vOut,scaleV,offsetVOptional)$$

  If cacheModeOptional is contiguous:

  $$kCacheRef[i][indice[i]]=quantK[i]$$ $$vCacheRef[i][indice[i]]=quantV[i]$$

  If cacheModeOptional is page:

  $$kCacheRefView=kCacheRef.view(-1,kCacheRef[-2],kCacheRef[-1])$$ $$vCacheRefView=vCacheRef.view(-1,vCacheRef[-2],vCacheRef[-1])$$ $$kCacheRefView[indices[i]]=quantK[i]$$ $$vCacheRefView[indices[i]]=quantV[i]$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

• [object Object]
• [object Object]

[object Object]

• Parameters

  [object Object] (aclTensor*, compute input): input x used for splitting in the formula, aclTensor on the device. The shape is [B, S, H] or [B, H]. H = (Nq + Nkv + Nkv) x D. The data type can be FLOAT16, INT32, or BFLOAT16. are supported. The supports ND. The shape can only be 2D or 3D.

  • [object Object] (aclTensor*, compute input): input cos used for position encoding in the formula, aclTensor on the device. If x is 3D, the shape is [B, S, 1, D]. If x is 2D, the shape is [B, D]. The data type can be FLOAT16 or BFLOAT16, which should be the same as that of sin. are supported. The supports ND. The shape can only be 2D or 4D.
  • [object Object] (aclTensor*, compute input): input sin used for position encoding in the formula, aclTensor on the device. If x is 3D, the shape is [B, S, 1, D]. If x is 2D, the shape is [B, D]. The data type can be FLOAT16 or BFLOAT16, which should be the same as that of cos. are supported. The supports ND. The shape can only be 2D or 4D.
  • [object Object] (aclTensor*, compute input): input kCacheRef used for caching k in the formula, aclTensor on the device, shape [C_1, C_2, Nkv, D], and INT8 data type. are not supported. The supports ND. The shape supports only 4D.
  • [object Object] (aclTensor*, compute input): input vCacheRef used for caching v in the formula, aclTensor on the device. The shape is [C_1, C_2, Nkv, D], and the data type can be INT8. are not supported. The supports ND. The shape supports only 4D.
  • [object Object] (aclTensor*, compute input): input indices indicating the token location information of Kvcache in the formula, aclTensor on the device. When cache_mode is page and x is 3D, the shape is [B*S]. Otherwise, the shape is [B]. The data type can be INT32. are supported. The supports ND. The shape can only be 1D or 2D.
  • [object Object] (aclTensor*, compute input): scaleK in the formula, used to quantize the scale factor of k. It is an aclTensor on the device, with shape [Nkv, D] and data type FLOAT. are supported. The supports ND. The shape supports only 2D.
  • [object Object] (aclTensor*, compute input): scaleV in the formula, used to quantize the scale factor of v. It is an aclTensor on the device, with shape [Nkv, D] and data type FLOAT. are supported. The supports ND. The shape supports only 2D.
  • [object Object] (aclTensor*, compute input): offsetKoptional in the formula, used to quantize the offset factor of k. It is an aclTensor on the device, with shape [Nkv, D] and data type FLOAT. are supported. The supports ND. The shape supports only 2D.
  • [object Object] (aclTensor*, compute input): offsetVoptional in the formula, used to quantize the offset factor. It is an aclTensor on the device, with shape [Nkv, D] and data type FLOAT. are supported. The supports ND. The shape supports only 2D.
  • [object Object] (aclTensor*, compute input): weightScaleoptional input in the formula, weight scale factor for dequantization. It is an aclTensor on the device, with shape [H] and data type FLOAT. are supported. The supports ND. The shape supports only 1D.
  • [object Object] (aclTensor*, compute input): activationScaleOptional input in the formula, activation scale factor for dequantization. It is an aclTensor on the device. If x is 3D, the shape is [B*S]. If x is 2D, the shape is [B]. The data type can be FLOAT. are supported. The supports ND. The shape supports only 1D.
  • [object Object] (aclTensor*, compute input): biasOptional input in the formula, bias for dequantization. It is an aclTensor on the device. The shape is [H], and the data type can be FLOAT, FLOAT16 (HALF), INT32, or BFLOAT16. are supported. The supports ND. The shape supports only 1D.
  • [object Object] (aclIntArray *, compute input): aclIntArray on the host. The data type is int array. The size is 3, and the value is [Nq D, Nkv D, Nkv * D]. It indicates the length of the input qkv to be split.
  • [object Object] (char*, compute input): expression string on the host. It indicates the supported quantization type. Currently, only [object Object] is supported.
  • [object Object] (char*, compute input): expression string on the host. It indicates the supported data format. Currently, only [object Object] is supported.
  • [object Object] (bool, compute input): Boolean value of the expression on the host. It indicates whether to output [object Object] and [object Object].
  • [object Object] (char*, compute input): expression string on the host. It indicates the update mode of [object Object]. Currently, only [object Object] and [object Object] are supported. The default value is [object Object].
  • [object Object] (aclTensor*, compute output): output [object Object] in the formula, which indicates the processed [object Object]. It is an aclTensor on the device. If [object Object] is a 3D tensor, the shape is [B, S, Nq, D]. If [object Object] is a 2D tensor, the shape is [B, Nq, D]. The data type can be FLOAT16 or BFLOAT16, and must be the same as that of [object Object]. are not supported. The can be ND.
  • [object Object] (aclTensor*, compute output): output [object Object] in the formula, which indicates the processed [object Object]. It is an aclTensor on the device. When [object Object] is false, [object Object] is empty. Otherwise, when [object Object] is a 3D tensor, the shape is [B, S, Nkv, D]. When [object Object] is a 2D tensor, the shape is [B, Nkv, D]. Has the same data type as [object Object]. are not supported. The can be ND.
  • [object Object] (aclTensor*, compute output): output [object Object] in the formula, which indicates the processed [object Object]. It is an aclTensor on the device. When [object Object] is false, [object Object] is empty. Otherwise, when [object Object] is a 3D tensor, the shape is [B, S, Nkv, D]. When [object Object] is a 2D tensor, the shape is [B, Nkv, D]. Has the same data type as [object Object]. are not supported. The can be ND.
  • [object Object] (uint64_t*, output): size of the workspace to be allocated on the device.
  • [object Object] (aclOpExecutor**, output): operator executor, containing the operator computation process.

• Returns

  aclnnStatus: status code. For details, see .

[object Object]

• Parameters

  • [object Object] (void*, input): start address of the workspace to be allocated on the device.
  • [object Object] (uint64_t, input): size of the workspace to be allocated on the device, which is obtained by the first-phase API aclnnDequantRopeQuantKvcacheGetWorkspaceSize.
  • [object Object] (aclOpExecutor *, input): operator executor, containing the operator computation process.
  • [object Object] (aclrtStream, input parameter): stream for executing the task.

• Returns

  aclnnStatus: status code. For details, see .

[object Object]

1. Deterministic computing:

   • [object Object] defaults to a deterministic implementation.

2. When [object Object] is set to contiguous, the 0th dimension of [object Object] is greater than that of x, and the value of indices is greater than or equal to 0 and less than or equal to the value of the 1st dimension of [object Object] (s in the [b, s, n, d] format) minus the value of the 1st dimension of [object Object]. When [object Object] is set to page, the value of inidces is greater than or equal to 0, less than the value of the 0th dimension multiplied by the value of the first dimension of [object Object], and is unique.

3. The last axis of [object Object] is less than or equal to 4096 and is 64-pixel aligned.

4. If [object Object] is not of type INT32, the data types of [object Object], [object Object], and [object Object] are the same as those of the outputs [object Object], [object Object], and [object Object]. In this case, [object Object], [object Object], and [object Object] do not take effect. If x is of type INT32, the data types of [object Object] and [object Object] are the same as those of the outputs [object Object], [object Object], and [object Object]. In this case, [object Object] is mandatory, and [object Object] and [object Object] are optional ([object Object] does not need to be the same as other input types).

[object Object]

The following example is for reference only. For details, see .