Documentation
Download
文档中心
AddScopeFeature

aclnnMatmulCompress

Applicable Products

ProductSupported
Ascend 950PR/Ascend 950DT×
Atlas A3 training products/Atlas A3 inference products×
Atlas A2 training products/Atlas A2 inference products×
Atlas 200I/500 A2 inference products×
Atlas inference products√
Atlas training products×

Function

  • Description: When performing l@r Matrix Multiplication, you can first use the msModelSlim tool to perform lossless compression on the r matrix to reduce its memory footprint, and then use this API to complete lossless decompression, Matrix Multiplication, and Dequantization.

  • Formula:

x2_unzip=unzip(x2,compressIndex)result=x1@x2_unzip+biasx2\_unzip = unzip(x2, compressIndex)\\ result=x1 @ x2\_unzip + bias

Where x2 represents the one-dimensional data of the r matrix compressed by the msModelSlim tool, compressIndex represents information related to the compression algorithm, and x2_unzipx2\_unzip is the data after lossless decompression performed internally by this API (consistent with the original r matrix data). For detailed usage examples of compression and calling this API, see Example.

Function Prototype

Each operator is divided into a two-phase API. The "aclnnMatmulCompressGetWorkspaceSize" API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the "aclnnMatmulCompress" API is called to perform the computation.

aclnnStatus aclnnMatmulCompressGetWorkspaceSize(
  const aclTensor* x,
  const aclTensor* weight,
  const aclTensor* bias,
  const aclTensor* compressIndex,
  aclTensor*       out,
  uint64_t*        workspaceSize,
  aclOpExecutor**  executor)
aclnnStatus aclnnMatmulCompress(
  void*          workspace,
  uint64_t       workspaceSize,
  aclOpExecutor* executor,
  aclrtStream    stream)

aclnnMatmulCompressGetWorkspaceSize

  • Parameters

    Parameter Input/Output Description Instruction Data Type Data Format Dimension (Shape) Non-contiguous Tensor
    x Input Represents the left input of matrix multiplication, i.e., matrix x1 in the formula. - FLOAT16 ND 2 √
    weight Input Represents the right input of the compressed matrix multiplication, i.e., matrix $x2\_unzip$ in the formula, which is the input compressed by the weight_compression module of the msModelSlim tool. - FLOAT16 ND 1 √
    bias Input Represents the bias input, i.e., the matrix bias in the formula. The data type supports only FLOAT. A null pointer can be passed. The shape supports only (1, n) or (n), where n is the n in the output shape (m, n). - FLOAT ND 2 √
    compressIndex Input Represents the compressed index table for the right input of matrix multiplication. Obtained through the msModelSlim tool in the example. INT8 ND 1 √
    out Output Output tensor. Obtained through the msModelSlim tool in the example. FLOAT16 ND 2 -
    workspaceSize Output Returns the workspace size to be allocated on the Device side. - - - - -
    executor Output Returns the operator executor, which contains the operator computation flow. - - - - -

  • Return Value

    aclnnStatus: return code. For details, see aclnn Return Code.

    The first-phase API performs input parameter validation and returns an error in the following scenarios:

    Return Value Error Code Description
    ACLNN_ERR_PARAM_NULLPTR 161001 The input x, weight, or out is a null pointer.
    ACLNN_ERR_PARAM_INVALID 161002 The data type or data format of x, weight, or out is not within the supported range.

aclnnMatmulCompress

  • Parameters

    Parameter Input/Output Description
    workspace Input Address of the workspace memory applied for on the Device side.
    workspaceSize Input Size of the workspace applied for on the Device side, obtained through the first-phase API aclnnMatmulCompressGetWorkspaceSize.
    executor Input Operator executor that contains the operator computation flow.
    stream Input Specifies the stream for executing the task.

  • Return Value

    aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

  • Deterministic description:

    • Atlas training products, Atlas inference products: aclnnMatmulCompress defaults to a deterministic implementation.

Example

Prepare the data before compression.

The following example shows how to generate input data using the gen_data.py script, for reference only:

```python
import numpy as np
import os
import sys
from numpy import random

def write2file(data, path):
  with open(path, 'wb') as f:
      data.tofile(f)

if not os.path.exists("./data"):
    os.mkdir("./data")

if len(sys.argv) != 4:
  print("Usage: python gen_data.py m k n")
  sys.exit(1)

m = int(sys.argv[1])
k = int(sys.argv[2])
n = int(sys.argv[3])

if m <= 0 or k <= 0 or n <= 0:
  print("Error: m, k and n must be positive integers.")
  sys.exit(1)

# Randomly generate matrix mat1 with shape (m, k).
mat1 = random.randn(m, k).astype(np.float16)
write2file(mat1, "./data/mat1.bin")

# Randomly generate matrix mat2 with shape (n, k).
mat2 = random.randint(0, 100, size=(n, k)).astype(np.float16)
mat2 = np.transpose(mat2, (1, 0)).copy()
mat2 = mat2.view(np.int8)
np.save("./data/weight.npy", {'weight': mat2})
os.chmod("./data/weight.npy", 0o0640)

# Generate the output.
output = np.random.randn(m, n).astype(np.float16)
write2file(output, "./data/output.bin")

# Generate the bias.
bias = random.randn(n).astype(np.float32)
write2file(bias, "./data/bias.bin")
```

Execute gen_data.py, assuming the shape parameters of mat1 and mat2 are m=512, k=1024, and n=1024.

```shell
python3 gen_data.py 512 1024 1024
```

  1. Preprocess the data.

    The original weight is processed by the msModelSlim compression tool to generate the compressed x2, compressIndex, and compressInfo: Before using the following APIs, compile the msModelSlim compression tool in the CANN package. For details, refer to the README.md in the msmodelslim/pytorch/weight_compression directory of the Gitee msit repository.

    from msmodelslim.pytorch.weight_compression import CompressConfig, Compressor

compress_config = CompressConfig(do_pseudo_sparse=False, sparse_ratio=1)
compressor = Compressor(compress_config, weight_path=weight_path)

compress_weight, compress_index, compress_info = compressor.run()
# Compressed weight, corresponding to x2 of the aclnnMatmulCompressGetWorkspaceSize API.
compressor.export(compress_weight, './data/weight')
# Index of the compressed weight, corresponding to the compressIndex parameter of the aclnnMatmulCompressGetWorkspaceSize API.
compressor.export(compress_index, './data/index')

  2. Call the aclnn API for computation.

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include <acl/acl.h>
#include <aclnnop/aclnn_matmul_compress.h>
#include <fstream>
#include <unistd.h>
#include <sys/stat.h>
#include <stdio.h>
#include <cstdlib>
#include <string>

#define CHECK_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      return_expr;                   \
    }                                \
  } while (0)

#define LOG_PRINT(message, ...)     \
  do {                              \
    printf(message, ##__VA_ARGS__); \
  } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
  int64_t shapeSize = 1;
  for (auto i : shape) {
    shapeSize *= i;
  }
  return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream) {
  // Boilerplate: resource initialization.
  auto ret = aclInit(nullptr);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
  ret = aclrtSetDevice(deviceId);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
  ret = aclrtCreateStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
  return 0;
}

int ReadBinFileNNop(std::string filePath, void* buffer, size_t bufferSize)
{
    struct stat sBuf;
    int fileStatus = stat(filePath.data(), &sBuf);
    CHECK_RET(fileStatus == ACL_SUCCESS, LOG_PRINT("Failed to get file %s\n", filePath); return -1);

    std::ifstream file;
    file.open(filePath, std::ios::binary);
    CHECK_RET(file.is_open(), LOG_PRINT("Open file failed.\n"); return -1);

    file.seekg(0, file.end);
    uint64_t binFileBufferLen = file.tellg();
    CHECK_RET(binFileBufferLen > 0,
        std::cout<<"File size is 0.\n";
        file.close();
        return -1);

    file.seekg(0, file.beg);
    file.read(static_cast<char *>(buffer), binFileBufferLen);
    file.close();
    return ACL_SUCCESS;
}

int CreateAclTensor(std::string filePath, const std::vector<int64_t>& shape, int typeSize,
                    void** deviceAddr, aclDataType dataType, aclTensor** tensor) {
  auto size = GetShapeSize(shape) * typeSize;
  // Call aclrtMalloc to allocate device-side memory.
  auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);

  // Call aclrtMallocHost to allocate host-side memory.
  void* binBufferHost = nullptr;
  ret = aclrtMallocHost(&binBufferHost, size);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMallocHost failed. ERROR: %d\n", ret); return ret);

  // Read the file.
  ret = ReadBinFileNNop(filePath, binBufferHost, size);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("ReadBinFileNNop failed. ERROR: %d\n", ret); return ret);

  // Call aclrtMemcpy to copy data from the host side to the device memory.
  ret = aclrtMemcpy(*deviceAddr, size, binBufferHost, size, ACL_MEMCPY_HOST_TO_DEVICE);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

  // Compute the strides of the contiguous tensor.
  std::vector<int64_t> strides(shape.size(), 1);
  for (int64_t i = shape.size() - 2; i >= 0; i--) {
    strides[i] = shape[i + 1] * strides[i + 1];
  }

  // Call the aclCreateTensor API to create an aclTensor.
  *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
  return 0;
}

int main(int argc, char* argv[]) {
  // 1. (boilerplate) Initialize the device/stream. Refer to the acl API manual.
  // Fill in deviceId based on the actual device.
  int32_t deviceId = 0;
  aclrtStream stream;
  auto ret = Init(deviceId, &stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

  if (argc != 6) {
    std::cerr << "Error: Invalid number of arguments. Usage: <program> m k n wCompressedSize indexSize" << std::endl;
    return -1;
  }

  // 2. Construct inputs and outputs. Custom construction is required based on the API.
  int m = atoi(argv[1]);
  int k = atoi(argv[2]);
  int n = atoi(argv[3]);
  // wShape is the size of the compressed right matrix data.
  int wCompressedSize = atoi(argv[4]);
  // indexShape is the size of the compressed index data.
  int indexSize = atoi(argv[5]);

  if (m <= 0 || k <= 0 || n <= 0 || wCompressedSize <= 0 || indexSize <= 0) {
    std::cerr << "Error: m, k, n, wCompressedSize and indexSize must be positive integers." << std::endl;
    return -1;
  }

  std::vector<int64_t> mat1Shape = {m, k};
  std::vector<int64_t> mat2CompressedShape = {wCompressedSize};
  std::vector<int64_t> indexShape = {indexSize};
  std::vector<int64_t> biasShape = {n};
  std::vector<int64_t> outputShape = {m, n};

  void* mat1DeviceAddr = nullptr;
  void* mat2CompressedDeviceAddr = nullptr;
  void* indexDeviceAddr = nullptr;
  void* biasDeviceAddr = nullptr;
  void* outputDeviceAddr = nullptr;

  aclTensor* mat1 = nullptr;
  aclTensor* mat2Compressed = nullptr;
  aclTensor* index = nullptr;
  aclTensor* bias = nullptr;
  aclTensor* output = nullptr;

  std::string rootPath = "./data/";

  // Create the mat1 aclTensor.
  std::string mat1FilePath = rootPath + "mat1.bin";
  ret = CreateAclTensor(mat1FilePath, mat1Shape, sizeof(uint16_t), &mat1DeviceAddr, aclDataType::ACL_FLOAT16, &mat1);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Create mat1 tensor failed. ERROR: %d\n", ret); return ret);
  // Create mat2Compressed aclTensor.
  std::string mat2FilePath = rootPath + "weight/weight.dat";
  ret = CreateAclTensor(mat2FilePath, mat2CompressedShape, sizeof(int8_t), &mat2CompressedDeviceAddr,
                        aclDataType::ACL_FLOAT16, &mat2Compressed);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Create mat2 tensor failed. ERROR: %d\n", ret); return ret);
  // Create index aclTensor.
  std::string indexFilePath = rootPath + "index/weight.dat";
  ret = CreateAclTensor(indexFilePath, indexShape, sizeof(int8_t), &indexDeviceAddr, aclDataType::ACL_INT8, &index);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Create index tensor failed. ERROR: %d\n", ret); return ret);
  // Create bias aclTensor.
  std::string biasFilePath = rootPath + "bias.bin";
  ret = CreateAclTensor(biasFilePath, biasShape, sizeof(float), &biasDeviceAddr, aclDataType::ACL_FLOAT, &bias);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Create bias tensor failed. ERROR: %d\n", ret); return ret);
  // Create out aclTensor.
  std::string outputFilePath = rootPath + "output.bin";
  ret = CreateAclTensor(outputFilePath, outputShape, sizeof(uint16_t), &outputDeviceAddr, aclDataType::ACL_FLOAT16, &output);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Create output tensor failed. ERROR: %d\n", ret); return ret);

  int32_t offsetX = 0;

  // 3. Call the CANN operator library API. Replace with the specific API name.
  uint64_t workspaceSize = 0;
  aclOpExecutor* executor;
  // Call the first-phase API of aclnnMatmulCompress.
  ret = aclnnMatmulCompressGetWorkspaceSize(mat1, mat2Compressed, bias, index, output, &workspaceSize, &executor);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnMatmulCompressGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
  // Allocate device memory based on the workspaceSize computed by the first-phase API.
  void* workspaceAddr = nullptr;
  if (workspaceSize > 0) {
    ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
  }
  // Call the second-phase API of aclnnMatmulCompress.
  ret = aclnnMatmulCompress(workspaceAddr, workspaceSize, executor, stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnMatmulCompress failed. ERROR: %d\n", ret); return ret);

  // 4. (Boilerplate) Synchronize and wait for task execution to complete.
  ret = aclrtSynchronizeStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

  // 5. Obtain the output value and copy the result from device memory to host memory. Modify this based on the specific API definition.
  auto size = GetShapeSize(outputShape);
  std::vector<float> resultData(size, 0);
  ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outputDeviceAddr,
                    size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
  for (int64_t i = 0; i < size; i++) {
    LOG_PRINT("result[%ld] is: %f\n", i, resultData[i]);
  }

  // 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
  aclDestroyTensor(mat1);
  aclDestroyTensor(mat2Compressed);
  aclDestroyTensor(index);
  aclDestroyTensor(bias);
  aclDestroyTensor(output);

  // 7. Release hardware resources. Modify based on the specific API definition.
  aclrtFree(mat1DeviceAddr);
  aclrtFree(mat2CompressedDeviceAddr);
  aclrtFree(indexDeviceAddr);
  aclrtFree(biasDeviceAddr);
  aclrtFree(outputDeviceAddr);
  if (workspaceSize > 0) {
    aclrtFree(workspaceAddr);
  }
  aclrtDestroyStream(stream);
  aclrtResetDevice(deviceId);
  aclFinalize();
  return 0;
}
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details