Overview of Matmul Performance Optimization Strategies

This section provides a series of performance tuning cases for operators involving Matmul computation. You can refer to the optimization methods and ideas in the cases and apply them to your specific scenarios. The cases are classified into five categories. The following table describes each category of cases. For details, see the following sections.

• Tiling optimization

  Table 1 Overview of tiling optimization strategies

  Category	Application Scenario	Case
  Tiling optimization: Optimize the strategy for tiling cores and basic blocks.	Large-shape scenarios with enough data size	Tiling Strategy for Matmul Operator Optimization

• Parallelism optimization

  Table 2 Overview of parallelism optimization strategies

  Category	Application Scenario	Case
  Inter-core task parallelism: Properly allocate data to different cores to execute tasks.	Scenarios where the K-axis of the matrix is large and the M-axis and N-axis are smaller than the K-axis.	Matmul High-Level API Enabling K-axis Tiling of Matrix Data in Multi-core Parallel Computation
  Inter-core data access parallelism: Optimize the multi-core data parallel access mechanism, for example, optimize the address access conflicts of the same memory data in multi-core scenarios, to improve the multi-core data access efficiency.	Scenarios where Matmul is executed on multiple cores, the K-axis of the input matrix is large, and the K-axis is not fully loaded.	Matmul High-Level API Enabling Multi-core K-axis Staggered Access to Device Memory
  Intra-core pipeline parallelism: Different instruction queues can be executed independently and in parallel, which can be used to optimize intra-core pipeline parallelism.	The MMAD pipeline and FIXPIPE pipeline of the operator are executed in serial mode. The synchronization waiting time accounts for a large proportion of the total execution time of the operator. MTE2 Bound and the MTE2 pipeline are executed in serial mode with other pipelines.	Matmul High-Level API Enabling UnitFlag Matmul High-Level API Enabling NBuffer33 Template

• Memory optimization

  Table 3 Overview of memory optimization strategies

  Category	Application Scenario	Case
  Memory sharing and reuse: Reduce the overhead caused by repeated data movement through buffer sharing and cache reuse.	In the MIX scenario, the GM addresses of matrix A or matrix B of multiple AIVs are the same, and the matrix A or matrix B reused by multiple AIVs are fully loaded on L1 Buffer.	Matmul High-Level API Enabling IBShare Template for Sharing Matrix A and Matrix B Data Matmul High-Level API Enabling IBShare Template for Sharing Matrix B Data
  Memory alignment: Ensure that the processed data meets specific alignment requirements. Use different data movement strategies for unaligned data to improve the data movement efficiency.	Scenarios where the axis in the input matrix is not 256-byte aligned and the data size is large.	ND2NZ Format Conversion on AIVs

• Scalar optimization

  Table 4 Overview of Scalar optimization strategies

  Category	Application Scenario	Case
  Static tiling: Complete Matmul tiling computation during kernel compilation. Convert variables into constants and spread them to the system to reduce Scalar computations and improve performance.	A large number of Scalar computations are performed during Matmul initialization, affecting the instruction header overhead. A large number of Scalar computations are performed between Matmul iterations, blocking the MTE2 pipeline.	Matmul High-Level API Enabling Full Static Tiling
  CUBE_ONLY: Reduce the extra Scalar overhead caused by the message processing mechanism.	Compared with the MIX mode, the Vector computation is not performed, and only Cube computation is performed.	Matmul High-Level API Enabling CUBE_ONLY

• Movement optimization

  Table 5 Overview of movement optimization strategies

  Category	Application Scenario	Case
  Movement throughput optimization: Properly control the size of the data block to be moved to improve the bandwidth utilization and movement efficiency.	Large-shape scenarios with a large number of MTE2 cyclic movements. Scenarios where the size of input and output data exceeds the L2 cache size.	Matmul High-Level API Enabling MDL Template Matmul High-Level API Enabling L2 Cache Tiling
  Preloading movement: Preload the data blocks to be moved to reduce the gap between pipelines.	Scenarios where the MTE2 pipeline gap is large and the value of M or N is large.	Matmul High-Level API Enabling MTE2 Preload