MatmulPolicy

Applicability

Function Usage

• NBuffer33MatmulPolicy (NBuffer33 template policy)

  The computation result of a matrix multiplication instruction is a matrix block of baseM × baseN. The matrix block is called a base block. The matrix A computed by a single core is divided into 3 × 3 base blocks. All the 3 × 3 base blocks of matrix A are loaded and stored in L1 Buffer. Each time, matrix multiplication is performed between the 3 × 3 base blocks of matrix A and the 3 × 1 base blocks of matrix B. At the same time, DoubleBuffer loads the 3 × 1 base blocks of matrix B required for the next computation in parallel until the matrix multiplication computation in the singleCoreN direction is complete. The following figure shows the NBuffer33 template policy. In the figure, singleCoreM, singleCoreN, and singleCoreK indicate the shape sizes of matrices A and B in a single core. Matrix A computed in a single core is divided into 3 × 3 base blocks, which are all loaded into L1 Buffer. These base blocks are multiplied by the 3 × 1 base blocks of matrix B each time.

  Figure 3 NBuffer33 template policy
  

• MatmulWithScalePolicy (MxMatmul template policy)

  This policy implements matrix multiplication with quantization coefficients, where both the left and right matrices have corresponding quantization coefficient matrices (left quantization coefficient matrix scaleA and right quantization coefficient matrix scaleB). The left quantization coefficient matrix is multiplied by the left matrix, and the right quantization coefficient matrix is multiplied by the right matrix. The products of these multiplications are then subjected to matrix multiplication with each other.

  Figure 4 MxMatmul template policy

  

Restrictions

• TrianUpperMatmulPolicy supports only the Norm template and MDL template.
• TrianLowerMatmulPolicy supports only the Norm template and MDL template.
• NBuffer33MatmulPolicy:
  • Currently, only the MDL template is supported.
  • The logical memory positions of matrices A and B support only TPosition::GM.
  • The MIX mode (including cube computation and vector computation) is not supported. Only the CUBE_ONLY mode (including only cube computation) is supported.
  • Only the IterateAll API can be used to obtain the computation result matrix C of Matmul.
  • The values of stepM, stepKa, and stepKb are less than or equal to 3, and the following condition is met: stepKa = stepKb = ceil(singleCoreK/baseK).
  • The sum of the base block size of matrix A (fully loaded) and the base block size of matrix B (loaded) does not exceed the size of L1 Buffer.
  • Before calling GetTiling to generate tiling parameters, you must call SetMatmulConfigParams to set scheduleTypeIn to ScheduleType::N_BUFFER_33 to enable the tiling generation logic of the NBuffer33 template policy.
• MatmulWithScalePolicy:
  • Supported only on the Atlas 350 Accelerator Card.
  • Currently, only the Norm template and MDL template are supported.
• SplitMMatmulPolicy:
  • Supported only on the Atlas 350 Accelerator Card.
  • Only matrix C can be output to Unified Buffer.
  • The IBSHARE parameter in MatmulType of matrices A and B must be set to true.

Example

The default template policy MatmulPolicy is the default value of the template parameter. The following describes how to use TrianUpperMatmulPolicy and TrianLowerMatmulPolicy.

• Example of using TrianUpperMatmulPolicy
  For details about the complete operator sample, see operator sample of using TrianUpperMatmulPolicy and TrianLowerMatmulPolicy.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

  #include "lib/matmul_intf.h" typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, half> aType; typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, half> bType; typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, float> cType; typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, float> biasType; // Input TrianUpperMatmulPolicy when defining Matmul. AscendC::Matmul<aType, bType, cType, biasType, CFG_NORM, MatmulCallBackFunc<nullptr, nullptr, nullptr>, AscendC::Impl::Detail::TrianUpperMatmulPolicy> mm; // Perform the regular Matmul computation and output the result in upper triangular format. TPipe pipe; TCubeTiling tiling; REGIST_MATMUL_OBJ(&pipe, GetSysWorkSpacePtr(), mm, &tiling); mm.SetTensorA(gmA, isTransposeA); mm.SetTensorB(gmB, isTransposeB); if (tiling.isBias) { mm.SetBias(gmBias); } mm.IterateAll(gmC); mm.End();

• Example of using TrianLowerMatmulPolicy
  For details about the complete operator sample, see operator sample of using TrianUpperMatmulPolicy and TrianLowerMatmulPolicy.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

  #include "lib/matmul_intf.h" typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, half> aType; typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, half> bType; typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, float> cType; typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, float> biasType; // Input TrianLowerMatmulPolicy when defining Matmul. AscendC::Matmul<aType, bType, cType, biasType, CFG_NORM, MatmulCallBackFunc<nullptr, nullptr, nullptr>, AscendC::Impl::Detail::TrianLowerMatmulPolicy> mm; // Perform the regular Matmul computation and output the result in lower triangular format. TPipe pipe; TCubeTiling tiling; REGIST_MATMUL_OBJ(&pipe, GetSysWorkSpacePtr(), mm, &tiling); mm.SetTensorA(gmA, isTransposeA); mm.SetTensorB(gmB, isTransposeB); if (tiling.isBias) { mm.SetBias(gmBias); } mm.IterateAll(gmC); mm.End();

• Example of using NBuffer33MatmulPolicy
  For details about the complete operator sample, see operator sample of enabling the NBuffer33 template policy.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

  #define ASCENDC_CUBE_ONLY #include "lib/matmul_intf.h" typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, half> aType; typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, half> bType; typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, float> cType; typedef AscendC::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, float> biasType; // Input NBuffer33MatmulPolicy when defining Matmul. AscendC::Matmul<aType, bType, cType, biasType, CFG_MDL, MatmulCallBackFunc<nullptr, nullptr, nullptr>, AscendC::Impl::Detail::NBuffer33MatmulPolicy> mm; // Use the NBuffer33 logic to perform Matmul computation and output the result. TPipe pipe; TCubeTiling tiling; REGIST_MATMUL_OBJ(&pipe, GetSysWorkSpacePtr(), mm, &tiling); mm.SetTensorA(gmA, isTransposeA); mm.SetTensorB(gmB, isTransposeB); if (tiling.isBias) { mm.SetBias(gmBias); } mm.IterateAll(gmC); mm.End();