Introduction to Memory Pool

Memory Pool

A memory pool is a common technique used to manage and allocate memory of a device. The basic functions are as follows:

• A memory pool applies for a large block of contiguous memory space and divides the memory space into small blocks of multiple granularities.
• When memory is requested, a proper block is selected from idle blocks and directly allocated.
• If no proper block is found, the system performs operations such as memory reclamation and space expansion to obtain a proper block.
• When the memory is released, the used block is marked as idle and reclaimed to the memory pool.

Structure

• MemoryBlock:

  A block requires at least three elements: an index for positioning, a block size description, and a corresponding actual physical address.

  struct MemoryBlock {
    // blockId: memory block index, which is globally unique.
    int64_t blockId;
  
    // blockSize: block size, in bytes.
    size_t blockSize;
  
    // Physical address of the memory block.
    void *address = nullptr;
  };

• MemoryPool:

  Basic memory pool, which manages the memory of each device. Generally, the number of memory pools to be created is the same as the number of devices. Typically, the unordered_map data structure helps manage all blocks. It contains three class methods: AllocateBlock, FreeBlock, and GetBlockPtr, which allocate blocks, release blocks, and obtain memory addresses in blocks, respectively.

• MemoryManager:

  Memory management class, which is responsible for initializing the memory pools of all devices and maintaining the memory pools in an array. When different threads or processes access this class and call the method, the GetDeviceId method is used to obtain the device ID corresponding to the thread and return the corresponding method of the corresponding device memory pool.

Key Functions

• MemoryPool::AllocateBlock

  The method of allocating memory blocks consists of two steps:

  1. The system searches for a memory block of the required size in the free block. If the memory block is found, the system returns the memory block index and marks the memory block as used.
  2. If no memory block meets the requirements but the pre-requested memory has remaining space: The system 64-byte aligns the start address of the remaining memory space, rounds up the allocation size to an integer multiple of 32 plus 32 bytes, marks the memory block as used, and returns the index.
• MemoryPool::FreeBlock

  When a memory block is released, the memory is not actually released but marked as unused.

Using Memory Pool to Manage Workspace Memory

The goal of using a memory pool is to cut down on aclrtMalloc and aclrtFree calls. Here, the memory pool manages the workspace memory.

The Node class stores the size of the workspace that has been allocated and the corresponding block ID. When the size of the required workspace changes, if the required size is less than the existing space, no operation is performed. If the required size is greater than the existing space, the current memory block is released and a larger memory block is applied for.