host_mem_pool_cfg

Applicability

Product	Supported (√/x)
Atlas 350 Accelerator Card	x
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	x
Atlas inference product	x
Atlas training product	x

Note: For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Configures host memory pool parameters.

Prototype

host_mem_pool_cfg(host_mem_pool_cfg)

Parameters

Parameter	Data Type	Description
host_mem_pool_cfg	string	A string in JSON array format, including memory_size and page_shift.

Configuration Item	Optional/Mandatory	Description
memory_size	Mandatory	Size of the current memory pool, in bytes. The value is of the int type and is greater than 0.
page_shift	Optional	Offset of page_size, which is used to calculate the value of page_size. The memory is aligned to a multiple of page_size during memory allocation. You need to set the memory size based on the site requirements. The type is int. The value range is [10, 31). For example, if page_shift is set to 16, the value of page_size is 1<<16 = 65536. The default value is 16.

Configuration Item

Optional/Mandatory

Description

memory_size

Mandatory

Size of the current memory pool, in bytes. The value is of the int type and is greater than 0.

page_shift

Optional

Offset of page_size, which is used to calculate the value of page_size. The memory is aligned to a multiple of page_size during memory allocation. You need to set the memory size based on the site requirements.

The type is int. The value range is [10, 31).

For example, if page_shift is set to 16, the value of page_size is 1<<16 = 65536.

The default value is 16.

Example

from llm_datadist import LLMConfig
llm_config = LLMConfig()
llm_config.host_mem_pool_cfg= "{\"memory_size\": 18737418240, \"page_shift\": 16}"

Returns

In normal cases, no value is returned.

If a parameter is incorrect, a TypeError or ValueError may be thrown.

Restrictions

The maximum size of the host memory pool is 20 GB.

Parent topic: LLMConfig (to Be Deprecated)