[object Object][object Object][object Object]undefined[object Object]

• Description: To improve performance, the inference network passes the sin and cos inputs through a cache to perform the rotary position encoding calculation.

• Formula:

  1. mrope mode: The shape of the positions input is [3, numTokens]:
  $$cosSin[i] = cosSinCache[positions[i]]$$ $$cos, sin = cosSin.chunk(2, dim=-1)$$ $$cos0 = cos[0, :, :mropeSection[0]]$$ $$cos1 = cos[1, :, mropeSection[0]:(mropeSection[0] + mropeSection[1])]$$ $$cos2 = cos[2, :, (mropeSection[0] + mropeSection[1]):(mropeSection[0] + mropeSection[1] + mropeSection[2])]$$ $$cos = torch.cat((cos0, cos1, cos2), dim=-1)$$ $$sin0 = sin[0, :, :mropeSection[0]]$$ $$sin1 = sin[1, :, mropeSection[0]:(mropeSection[0] + mropeSection[1])]$$ $$sin2 = sin[2, :, (mropeSection[0] + mropeSection[1]):(mropeSection[0] + mropeSection[1] + mropeSection[2])]$$ $$sin= torch.cat((sin0, sin1, sin2), dim=-1)$$ $$queryRot = query[..., :rotaryDim]$$ $$queryPass = query[..., rotaryDim:]$$

  (1) rotate_half (GPT-NeoX style) computation formula:

  $$x1, x2 = torch.chunk(queryRot, 2, dim=-1)$$ $$o1[i] = x1[i] * cos[i] - x2[i] * sin[i]$$ $$o2[i] = x2[i] * cos[i] + x1[i] * sin[i]$$ $$queryRot = torch.cat((o1, o2), dim=-1)$$ $$query = torch.cat((queryRot, queryPass), dim=-1)$$

  (2) rotate_interleaved (GPT-J style) computation formula:

  $$x1 = queryRot[..., ::2]$$ $$x2 = queryRot[..., 1::2]$$ $$queryRot = torch.stack((o1, o2), dim=-1)$$ $$query = torch.cat((queryRot, queryPass), dim=-1)$$
  2. Rope mode: The shape of the positions input is [numTokens]:
  $$cosSin[i] = cosSinCache[positions[i]]$$ $$cos, sin = cosSin.chunk(2, dim=-1)$$ $$queryRot = query[..., :rotaryDim]$$ $$queryPass = query[..., rotaryDim:]$$

  (1) rotate_half (GPT-NeoX style) computation formula:

  $$x1, x2 = torch.chunk(queryRot, 2, dim=-1)$$ $$o1[i] = x1[i] * cos[i] - x2[i] * sin[i]$$ $$o2[i] = x2[i] * cos[i] + x1[i] * sin[i]$$ $$queryRot = torch.cat((o1, o2), dim=-1)$$ $$query = torch.cat((queryRot, queryPass), dim=-1)$$

  (2) rotate_interleaved (GPT-J style) computation formula:

  $$x1 = query\_rot[..., ::2]$$ $$x2 = query\_rot[..., 1::2]$$ $$queryRot = torch.stack((o1, o2), dim=-1)$$ $$query = torch.cat((queryRot, queryPass), dim=-1)$$

[object Object]

Each operator has calls. It is necessary to first call the [object Object] interface to obtain the required workspace size for computation and the executor that includes the operator's computation process. Then call the [object Object] interface to execute the computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • aclnnNormRopeConcatBackward defaults to deterministic implementation.

• [object Object], [object Object], and [object Object] only support 2-dimensional shape input.

• The data types of [object Object], [object Object], and [object Object] inputs need to be consistent.

• [object Object]: When the data type is BFLOAT16 or FLOAT16, it should be a multiple of 32; when the data type is FLOAT32, it should be a multiple of 16.

• [object Object]: Always less than or equal to headSize; when the data type is BFLOAT16 or FLOAT16, it should be a multiple of 32, and when the data type is FLOAT32, it should be a multiple of 16; in mrope mode, it should satisfy rotaryDim = mropeSection[0] + mropeSection[1] + mropeSection[2].

• The value of the input tensor positions should be less than the [object Object] of the 0th dimension of [object Object].

• [object Object] (in mrope mode): the value is limited to [16, 24, 24], and the value of [object Object] is 128.

[object Object]

The following example is for reference only. For details, see .