Basic API Porting Guide

Vector Computation

• By default, the 351x architecture does not support the subnormal function.

  Note: A subnormal floating-point number refers to a floating-point number whose exponent bits are all 0s and whose mantissa is not 0. It is used to represent a value smaller than the smallest normal number, avoiding underflow to 0. By default, the 351x architecture does not support subnormal floating-point numbers and will treat such numbers as 0 during computation.

  Compatibility solution: Set the config template parameters to configure the subnormal data computation mode. The software simulates the processing of subnormal data and uses methods such as precision extension to prevent subnormal floating-point numbers from underflowing to 0.

  Table 1 APIs and config parameters involving subnormal data

  Ascend C Basic API

  Compatibility Description

  Exp, Ln, Reciprocal, Sqrt, Rsqrt, Div

  The Ln API is used as an example. The algo parameter in the LnConfig structure is used to configure the subnormal data computation mode. The options of algo are as follows: LnAlgo::INTRINSIC and LnAlgo::PRECISION_1ULP_FTZ_TRUE: The result is computed using a single instruction, and all subnormal numbers are approximated to 0. LnAlgo::PRECISION_1ULP_FTZ_FALSE: Subnormal data computation is supported. The value of the default parameter DEFAULT_LN_CONFIG is as follows: 1 constexpr LnConfig DEFAULT_LN_CONFIG = { LnAlgo::INTRINSIC };

  Refer to the following code snippet:

  // Define the template parameter.
  constexpr AscendC::LnConfig CONFIG = {
      AscendC::LnAlgo::PRECISION_1ULP_FTZ_FALSE
  };
  template <typename T>
  __aicore__ inline void Compute(GM_ADDR dst, GM_ADDR src, uint32_t count)
  {
      AscendC::TPipe pipe;
      AscendC::GlobalTensor<T> srcGlobal;
      AscendC::GlobalTensor<T> dstGlobal;
      srcGlobal.SetGlobalBuffer((__gm__ T*)src);
      dstGlobal.SetGlobalBuffer((__gm__ T*)dst);
      AscendC::TQue<AscendC::TPosition::VECIN, 1> inQueue;
      AscendC::TQue<AscendC::TPosition::VECOUT, 1> outQueue;
      pipe.InitBuffer(inQueue, 1, count * sizeof(T));
      pipe.InitBuffer(outQueue, 1, count * sizeof(T));
      AscendC::LocalTensor<T> dstLocal = outQueue.AllocTensor<T>();
      AscendC::LocalTensor<T> srcLocal = inQueue.AllocTensor<T>();
      AscendC::DataCopy(srcLocal, srcGlobal, count);
      AscendC::SetFlag<AscendC::HardEvent::MTE2_V>(EVENT_ID0);
      AscendC::WaitFlag<AscendC::HardEvent::MTE2_V>(EVENT_ID0);
      // Call the basic API Ln and input the template parameter.
      AscendC::Ln<T, CONFIG>(dstLocal, srcLocal, count);
      AscendC::SetFlag<AscendC::HardEvent::V_MTE3>(EVENT_ID1);
      AscendC::WaitFlag<AscendC::HardEvent::V_MTE3>(EVENT_ID1);
      AscendC::DataCopy(dstGlobal, dstLocal, count);
      inQueue.FreeTensor(srcLocal);
  }

Data Movement

• The DataCopy API does not support the channel from the L1 buffer to the global memory.

  Note: The channel from the L1 buffer to the global memory is deleted from the hardware. Therefore, data cannot be directly moved from the L1 buffer to the global memory. The existing APIs do not support direct data movement from the L1 buffer to the global memory.

  Compatibility solution: In cube-only computation scenarios, allocate an additional identity matrix in the global memory, use the MMAD matrix multiplication to compute the data and output it to the L0C buffer, and then use FixPipe to move the data from the L0C buffer to the global memory. In scenarios where vector and cube computations are combined, data can be moved from the L1 buffer to the unified buffer and then to the global memory. The following uses the cube-only computation scenario as an example to describe the core operator process:

  1. Move matrix A from the global memory to the L1 buffer.

     __aicore__ inline void CopyGmToA1()
     {
         AscendC::LocalTensor<T> leftMatrix = inQueueA1.AllocTensor<T>();
         AscendC::Nd2NzParams intriParams1{1, 64, 128, 0, 128, 64, 1, 0};
         AscendC::DataCopy(leftMatrix, aGlobal, intriParams1);
         inQueueA1.EnQue(leftMatrix);
     }

  2. Move matrix B (an identity matrix) from the global memory to the L1 buffer.

     __aicore__ inline void CopyGmToB1()
     {
         AscendC::LocalTensor<U> rightMatrix = inQueueB1.AllocTensor<U>();
         AscendC::Nd2NzParams intriParams2{1, 128, 128, 0, 128, 128, 1, 0};
         AscendC::DataCopy(rightMatrix, bGlobal, intriParams2);
         inQueueB1.EnQue(rightMatrix);
     }

  3. Move matrix A from the L1 buffer to the L0A buffer.

     __aicore__ inline void Load2DA1ToL0A()
     {
         AscendC::LocalTensor<T> a1 = inQueueA1.DeQue<T>();
         AscendC::LocalTensor<T> a2 = inQueueA2.AllocTensor<T>();
         AscendC::LoadData2DParamsV2 loadDataParams;
         ...
         AscendC::LoadData(a2, a1, loadDataParams);
         ...
     }

  4. Move matrix B from the L1 buffer to the L0B buffer.

     __aicore__ inline void Load2DA1ToL0B()
     {
         AscendC::LocalTensor<U> b1 = inQueueB1.DeQue<U>();
         AscendC::LocalTensor<U> b2 = inQueueB2.AllocTensor<U>();
         ...
         AscendC::LoadData2DParamsV2 loadDataParams;
         ...
         AscendC::LoadData(b2, b1, loadDataParams);
         ...
     }

  5. Perform MMAD matrix computation and output the result to the L0C buffer.

     __aicore__ inline void Compute()
     {
         AscendC::MmadParams mmadParams;
         ...
         AscendC::LocalTensor<S> co1Local = inQueueCO1.AllocTensor<S>();
         AscendC::LocalTensor<T> a2 = inQueueA2.DeQue<T>();
         AscendC::LocalTensor<U> b2 = inQueueB2.DeQue<U>();
         AscendC::Mmad(co1Local, a2, b2, mmadParams);
         ...
     }

  6. Copy matrix C from the L0C buffer to the global memory through FixPipe.

     __aicore__ inline void CopyL0CToGm()
     {
         AscendC::LocalTensor<S> co1Local = inQueueCO1.DeQue<S>();
         AscendC::FixpipeParamsV220 fixpipeParams;
         ...
         AscendC::Fixpipe<S, S, AscendC::CFG_ROW_MAJOR>(cGlobal, co1Local, fixpipeParams);
         ...
     }

Matrix Computation

• The int4b_t data type is deleted from the cube unit.

  Note: Compared with the 220x architecture version, the cube unit of the 351x architecture version does not support the int4b_t data type. The related basic APIs include LoadData, Mmad, and LoadDataWithTranspose, which no longer support the int4b_t data type.

  Compatibility solution: Write a CV fused operator to cast int4b_t data to int8_t data on the vector core. Then, move the cast data to the L1 buffer through the unified buffer and perform MMAD computation. At the graph layer, you can add a Cast node before the operator to cast int4b_t data to int8_t data.

  1. Cast int4b_t data to int8_t data on the vector core. The cast data is stored in the new global memory space.

     __aicore__ inline void Unzip(AscendC::GlobalTensor<int8_t>& dstGlobalTensor,
                                  AscendC::GlobalTensor<int8_t>& srcGlobalTensor, uint32_t count,
                                  AscendC::TQue<AscendC::TPosition::VECIN, 1>& q1,
                                  AscendC::TQue<AscendC::TPosition::VECOUT, 1>& q2,
                                  AscendC::TQue<AscendC::TPosition::VECOUT, 1>& q3)
     {
         AscendC::LocalTensor<int8_t> srcLocalTensor = q1.AllocTensor<int8_t>();
         AscendC::LocalTensor<half> tmpTensor = q2.AllocTensor<half>();
         AscendC::LocalTensor<int8_t> dstLocalTensor = q3.AllocTensor<int8_t>();
         AscendC::DataCopy(srcLocalTensor, srcGlobalTensor, count);
         AscendC::LocalTensor<AscendC::int4b_t> int4SrcLocalTensor = srcLocalTensor.ReinterpretCast<AscendC::int4b_t>();
         uint32_t mask = count / sizeof(half);
         AscendC::SetFlag<AscendC::HardEvent::MTE2_V>(EVENT_ID0);
         AscendC::WaitFlag<AscendC::HardEvent::MTE2_V>(EVENT_ID0);
         AscendC::Cast<half, AscendC::int4b_t>(tmpTensor, int4SrcLocalTensor, AscendC::RoundMode::CAST_NONE, count * 2);
         AscendC::Cast<int8_t, half>(dstLocalTensor, tmpTensor, AscendC::RoundMode::CAST_CEIL, count * 2);
         AscendC::SetFlag<AscendC::HardEvent::V_MTE3>(EVENT_ID1);
         AscendC::WaitFlag<AscendC::HardEvent::V_MTE3>(EVENT_ID1);
         AscendC::DataCopy(dstGlobalTensor, dstLocalTensor, count * 2);
         q1.FreeTensor(srcLocalTensor);
         q2.FreeTensor(tmpTensor);
         q3.FreeTensor(dstLocalTensor);
     }

  2. Perform matrix computation on data of the int8_t type.

     template <class A_TYPE, class B_TYPE, class C_TYPE, class BIAS_TYPE>
     __aicore__ inline void MatMulKernel(AscendC::GlobalTensor<int8_t>& aGlobal, AscendC::GlobalTensor<int8_t>& bGlobal,
                                         AscendC::GlobalTensor<int32_t>& cGlobal, GM_ADDR tilingGM, GM_ADDR workspaceGM,
                                         int32_t isTransposeAIn, int32_t isTransposeBIn, AscendC::TPipe& que)
     {
         using A_T = typename A_TYPE::T;
         using B_T = typename B_TYPE::T;
         using C_T = typename C_TYPE::T;
         using BiasT = typename BIAS_TYPE::T;
         ...
         int offsetA = 0;
         int offsetB = 0;
         int offsetC = 0;
         int offsetBias = 0;
         ...
         auto gmA = aGlobal[offsetA];
         auto gmB = bGlobal[offsetB];
         auto gmC = cGlobal[offsetC];
         AscendC::MatmulImpl<A_TYPE, B_TYPE, C_TYPE, BIAS_TYPE, CFG_MDL> mm;
         mm.SetSubBlockIdx(0);
         mm.Init(&tiling, &que);
         mm.SetTensorA(gmA, isTransposeA);
         mm.SetTensorB(gmB, isTransposeB);
         mm.IterateAll(gmC);
     }

• The fractal format of the L0A buffer is changed from ZZ to ZN.

  Note: The involved APIs are LoadData, Mmad, and LoadDataWithTranspose.

  • In the 220x architecture, when matrix multiplication (A × B = C) is performed, the data formats of matrices A, B, and C are ZZ, ZN, and NZ, respectively. Matrices A, B, and C are located in the L0A buffer, L0B buffer, and L0C buffer, respectively.

    Matrix A: The row-major order is used in each fractal matrix and between fractal matrices. The fractal shape is 16 × (32 bytes/sizeof(AType)), and the size is 512 bytes.

    Matrix B: The column-major order is used in each fractal matrix, while the row-major order is used between fractal matrices. The fractal shape is (32 bytes/sizeof(BType)) × 16, and the size is 512 bytes.

    Matrix C: The row-major order is used in each fractal matrix, while the column-major order is used between fractal matrices. The fractal shape is 16 × 16, and the size is 256 elements.

    

  • In the 351x architecture, when matrix multiplication (A × B = C) is performed, the data formats of matrices A, B, and C are NZ, ZN, and NZ, respectively.

    Matrix A: The row-major order is used in each fractal matrix, while the column-major order is used between fractal matrices. The shape is 16 × (32 bytes/sizeof(AType)), and the size is 512 bytes.

    Matrix B: The column-major order is used in each fractal matrix, while the row-major order is used between fractal matrices. The fractal shape is (32 bytes/sizeof(BType)) × 16, and the size is 512 bytes.

    Matrix C: The row-major order is used in each fractal matrix, while the column-major order is used between fractal matrices. The fractal shape is 16 × 16, and the size is 256 elements.

    

  Compatibility solution: The 220x version is compatible in non-L0A buffer tiling scenarios. In L0A buffer tiling scenarios, perform adaptation based on the new fractal.

  In the 220x architecture, matrix computation requires the left matrix to be a ZZ fractal (NZ in 350x) and the right matrix to be a ZN fractal. Because the data fractal of the L1 buffer is NZ, moving the left matrix from the L1 buffer to the L0A buffer in the 220x architecture requires an additional conversion from the NZ fractal to the ZZ fractal. In the 351x architecture, no fractal conversion is required.

  

  The changes caused by fractal changes are mainly reflected in the process of moving data from the L1 buffer to the L0A buffer. The following code snippet shows the changes:

  __aicore__ inline void SplitA()
  {
      int srcOffset = 0;
      int dstOffset = 0;
      AscendC::LocalTensor<half> a1Local = inQueueA1.DeQue<half>();
      AscendC::LocalTensor<half> a2Local = inQueueA2.AllocTensor<half>();
      // Nz2Zz fractal conversion is performed when LoadData is called in the 220x architecture.
      // for (int i = 0; i < mBlocks; ++i) {
      //     AscendC::LoadData2DParams loadDataParams;
      //     loadDataParams.repeatTimes = kBlocks; // kBlocks indicates the number of half-type matrices with a width of 16 in the column direction.
      //     loadDataParams.srcStride = mBlocks; // mBlocks indicates the number of half-type matrices with a height of 16 in the row direction.
      //     loadDataParams.ifTranspose = false;
      //     AscendC::LoadData(a2Local[dstOffset], a1Local[srcOffset], loadDataParams);
      //     srcOffset += 16 * 16;
      //     dstOffset += k * 16;
      // }
      // In the 350x architecture, Nz2Zz fractal conversion is not required when LoadData is called. The corresponding movement parameters need to be modified.
      AscendC::LoadData2DParams loadDataParams;
       loadDataParams.repeatTimes = m * k / 512; // Number of small-z matrices.
      loadDataParams.srcStride = 1; // Interval between small-z matrices.
      loadDataParams.dstGap = 0;
      loadDataParams.ifTranspose = false;
      AscendC::LoadData(a2Local, a1Local, loadDataParams);
      
      inQueueA2.EnQue<half>(a2Local);
      inQueueA1.FreeTensor(a1Local);
  }

• The 4:2 structured sparsity function is deleted from the hardware architecture of the 351x architecture version.

  Note: LoadDataWithSparse is used to move the 512-byte dense weight matrix stored in the L1 buffer to the L0B buffer and read the 128-byte index matrix to implement sparsification of the dense matrix. Since the 351x architecture version does not support structured sparsity, LoadDataWithSparse is not applicable in this version. MmadWithSparse is responsible for performing the matrix multiplication and addition operations, where the right matrix B is a dense matrix and needs to be loaded by calling LoadDataWithSparse. Since the 351x architecture does not support LoadDataWithSparse, MmadWithSparse cannot be used in the 351x architecture version.

  Compatibility solution: Do not call LoadDatawithSparse to convert a dense matrix to a sparse matrix on the operator side. Use Mmad to perform normal dense matrix computation. For details about algorithms related to matrix sparsity, see the description of MmadWithSparse.

• The channels from the global memory to the L0A buffer/L0B buffer are deleted from the 351X architecture version.

  Note: The channels from the global memory to the L0A buffer/L0B buffer are deleted from the hardware. When LoadData is called, these channels are no longer supported.

  Compatibility solution: Split the data movement from the global memory to the L0A buffer/L0B buffer into two steps. First, move data from the global memory to the L1 buffer, and then from the L1 buffer to the L0A buffer/L0B buffer.

  The following uses the channel from the global memory to the L1 buffer and then to the L0A buffer as an example:

  1. Move matrix A from the global memory to the L1 buffer.

     __aicore__ inline void CopyGmToA1()
     {
         AscendC::LocalTensor<T> leftMatrix = inQueueA1.AllocTensor<T>();
         AscendC::Nd2NzParams intriParams1{1, 64, 128, 0, 128, 64, 1, 0};
         AscendC::DataCopy(leftMatrix, aGlobal, intriParams1);
         inQueueA1.EnQue(leftMatrix);
     }

  2. Move matrix A from the L1 buffer to the L0A buffer.

     __aicore__ inline void Load2DA1ToL0A()
     {
         AscendC::LocalTensor<T> a1 = inQueueA1.DeQue<T>();
         AscendC::LocalTensor<T> a2 = inQueueA2.AllocTensor<T>();
         AscendC::LoadData2DParamsV2 loadDataParams;
         ...
         AscendC::LoadData(a2, a1, loadDataParams);
         ...
     }

• Hardware instructions related to L0A buffer/L0B buffer initialization are deleted in the 351x architecture version.

  Note: The InitConstValue API initializes the LocalTensor at a specific storage location to a specific value. The L0A buffer and L0B buffer cannot be directly initialized.

  Compatibility solution: Initialize the L1 buffer first, and then use the LoadData API to move data from the L1 buffer to the L0A buffer and L0B buffer.

  The following uses the data channel from the global memory to the L1 buffer and then to the L0A buffer as an example:

  1. Initialize the L1 buffer.

     __aicore__ inline void InitConstA1()
     {
         AscendC::LocalTensor<T> leftMatrix = inQueueA1.AllocTensor<T>();
         AscendC::InitConstValue(leftMatrix, {1, static_cast<uint16_t>(m * k * sizeof(T) / 32), 0, 1});
         inQueueA1.EnQue(leftMatrix);
     }

  2. Call the LoadData API to move data from the L1 buffer to the L0A buffer.

     __aicore__ inline void Load2DA1ToL0A()
     {
         AscendC::LocalTensor<T> a1 = inQueueA1.DeQue<T>();
         AscendC::LocalTensor<T> a2 = inQueueA2.AllocTensor<T>();
         AscendC::LoadData2DParamsV2 loadDataParams;
         ...
         AscendC::LoadData(a2, a1, loadDataParams);
         inQueueA2.EnQue(a2);
         inQueueA1.FreeTensor(a1);
     }

System Variable Access

• CheckLocalMemoryIA is not supported, and related registers are deleted from the 351x architecture version.

  Note: CheckLocalMemoryIA monitors the unified buffer read and write operations within the specified range. If the unified buffer read and write operations within the specified range are monitored, an EXCEPTION error is reported. If the unified buffer read and write operations within the specified range are not monitored, no error is reported.

  Compatibility solution: This API is a debugging API and has no impact on functions.

Debugging API

• The L1 buffer does not support the printing of tensor information.

  Note: The channel from the L1 buffer to the global memory is deleted from the chip. Therefore, functions involving data movement from the L1 buffer to the global memory are not supported.

  Compatibility solution: This API is a debugging API and has no impact on functions.