[object Object]

[object Object][object Object]undefined[object Object]

• Description: Performs an optional dequantization on the input tensor (x), then splits it along the last axis into q, k, and vOut according to the specified sizeSplits. It applies Rotary Positional Embedding (RoPE) to q and k to generate qOut and kOut. Finally, kOut and vOut are quantized and updated into kCacheRef and vCacheRef based on the provided indices.

• Formula:

  $$dequantX = Dequant(x,weightScaleOptional,activationScaleOptional,biasOptional)$$ $$q,k,vOut = SplitTensor(dequantX,dim=-1,`sizeSplits`)$$ $$qOut,kOut = ApplyRotaryPosEmb(q,k,cos,sin)$$ $$quantK = Quant(kOut,scaleK,offsetKOptional)$$ $$quantV = Quant(vOut,scaleV,offsetVOptional)$$

  If cacheModeOptional is contiguous:

  $$kCacheRef[i][indice[i]]=quantK[i]$$ $$vCacheRef[i][indice[i]]=quantV[i]$$

  If cacheModeOptional is page:

  $$kCacheRefView=kCacheRef.view(-1,kCacheRef[-2],kCacheRef[-1])$$ $$vCacheRefView=vCacheRef.view(-1,vCacheRef[-2],vCacheRef[-1])$$ $$kCacheRefView[indices[i]]=quantK[i]$$ $$vCacheRefView[indices[i]]=quantV[i]$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

1. Deterministic computing:

   • [object Object] defaults to a deterministic implementation.

2. When cacheModeOptional is set to contiguous, the 0th dimension of kCacheRef is greater than the 0th dimension of x, and the indices data value is greater than or equal to 0 and less than or equal to the first dimension of vCacheRef (s in the [b, s, n, d] format) minus the first dimension of x.

3. When cacheModeOptional is set to page, the indices data value is greater than or equal to 0, less than the product of the 0th and 1st dimensions of kCacheRef, and is unique.

4. If the input x is not of type INT32, the data types of x, cos, and sin are the same as those of the outputs qOut, kOut, and vOut. In this case, activationScaleOptional, weightScaleOptional, and biasOptional do not take effect.

5. If the input x is of type INT32, the data types of cos and sin are the same as those of the outputs qOut, kOut, and vOut. In this case, weightScaleOptional is required, and activationScaleOptional and biasOptional are optional. (The data type of biasOptional does not need to be the same as that of other inputs.)

6. The last axis of [object Object] is less than or equal to 4096 and is 64-pixel aligned.

[object Object]

The following example is for reference only. For details, see .