Memory Management

ge.exec.atomicCleanPolicy

Sets whether to collectively clean up the memory occupied by all operators with the memset attribute (memset operators) on the network.

Parameter values:

• 0 (default): Enables collective cleanup.
• 1: Disables collective cleanup. Memory used by each memset operator is cleaned up separately. When the memset operators on the network occupy too much memory, you are advised to use this mode to reduce the memory usage. However, this may cause performance loss.

Configuration example:

{"ge.exec.atomicCleanPolicy", "0"};

Mandatory/Optional: optional

Effective level: session

ge.exec.disableReuseMemory

Enables memory reuse.

Parameter values:

• 1: memory reuse disabled
• 0 (default): memory reuse enabled

Configuration example:

{"ge.exec.disableReuseMemory", "0"};

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.exec.input_fusion_size

Sets the threshold for fusing and copying multiple discrete pieces of user input data during data movement from the host to the device. The value ranges from 0 to 33,554,432 bytes (32 MB) and defaults to 131,072 bytes (128 KB).

• If the size of input data is less than or equal to the threshold, the data is fused and then moved from the host to the device.
• If the size of input data is greater than the threshold or the threshold is 0 (the function is disabled), the data is not fused but is directly moved from the host to the device.

Assume there are 10 user inputs, including two 100 KB inputs, two 50 KB inputs, and the other inputs greater than 100 KB:

• ge.exec.input_fusion_size = 100 KB: The preceding four inputs are fused into 300 KB data for movement. The other six inputs are directly moved from the host to the device.
• ge.exec.input_fusion_size = 0 KB: This function is disabled and no inputs are fused. That is, the 10 inputs are directly moved from the host to the device.

This parameter takes effect only when the static graph is run asynchronously. That is, the API mentioned in RunGraphAsync is used to run the graph.

Configuration example:

{"ge.exec.input_fusion_size", "131072"};

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.exec.inputReuseMemIndexes

Sets whether to enable the memory reuse function of the input node of a graph. After the function is enabled, the memory of the input node can be reused as the intermediate memory required during model execution, reducing the memory peak.

The value is the index of the input node. If memory reuse is enabled for multiple input nodes, use commas (,) to separate multiple indexes. The index attribute of the input node is required, specifying the sequence number of the input. The index starts from 0.

Note:

• If memory reuse of the input node is enabled, ensure that the input memory size is 32-byte aligned.
• If the configured input index is greater than or equal to the value of the input count, index is invalid and the index configuration does not take effect.
• If memory reuse of the input node is enabled, the memory data of the input node will be overwritten. After the graph is run, the memory data of the input node cannot be used.

Configuration example:

{"ge.exec.inputReuseMemIndexes", "0,1,2"};

Mandatory/Optional: optional

Effective level: graph

ge.exec.reuseZeroCopyMemory

Enables zero-copy memory reuse.

In traditional AI inference, data is copied from the host memory to the device memory, and then sent back after the inference is complete. These host-to-device and device-to-host transfers are the major source of inference latency. Zero-copy eliminates intermediate copies by directly mapping user buffers to device memory for read/write operations.

By default, each model execution reallocates a zero-copy I/O memory area. When this option is enabled, GE independently plans the zero-copy memory blocks for model inputs and outputs and keeps them separate from other memory blocks. It reuses the area across multiple executions. This avoids repeated memory allocation and significantly reduces inference latency.

Parameter values:

• 0 (default): disabled. Zero-copy memory is not reused. A new allocation occurs for each execution.
• 1: enabled. Zero-copy memory is reused. The input and output I/O memory is reused across executions.

Configuration example:

{"ge.exec.reuseZeroCopyMemory", "0"};

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.exec.staticMemoryPolicy

Sets the memory allocation mode during network running.

Parameter values:

• 0 (default): dynamic memory allocation. Memory is dynamically allocated based on the actual size. Dynamic memory expansion is not supported.
• 2: dynamic memory expansion of static shapes. In training and online inference scenarios, this environment variable can be used to implement memory reuse between multiple graphs in the same session. That is, the memory required by the maximum graph is allocated. For example, if the memory required by the current graph exceeds the memory of the previous graph, the memory of the previous graph is directly released. The memory is reallocated based on the memory required by the current graph.
• 3: Only dynamic shape supports dynamic memory expansion, which solves the fragment problem during dynamic memory allocation and reduces the memory usage of the dynamic-shape network.
• 4: Both static and dynamic shapes support dynamic memory expansion.

Configuration example:

{"ge.exec.staticMemoryPolicy", "0"};

Mandatory/Optional: optional

Effective level: global and session

ge.exec.outputReuseInputMemIndexes

Explicitly declares that the specified output tensor and input tensor share the same memory address during model inference.

If you can determine the memory reuse relationship before graph build, you can use this parameter to notify the GE graph compiler, which will perform corresponding optimization during compilation. Typical scenarios are as follows:

Data > TensorMove > InplaceOp (in-place operator) > NetOutput, where Data and NetOutput are the model input and output, respectively. If the lifetime of Data and NetOutput meets the reuse conditions, the compiler automatically removes the intermediate TensorMove (memory copy operator) to avoid unnecessary memory copy operations and improve the model running performance.

Parameter values:

"input_index_0,output_index_0|input_index_1,output_index_1|..."

• Each pair of input_index_X and output_index_Y indicates that the input tensor with the input index X and the output tensor with the output index Y share the same memory address.
• Multiple pairs of indexes are separated by vertical bars (|).

Configuration example:

Assume that the model has 10 inputs and 10 outputs. If the configuration is as follows:

ge.exec.outputReuseInputMemIndexes = "0,1|6,8"

This indicates that the memory address of output index 1 is the same as that of input index 0, and the memory address of output index 8 is the same as that of input index 6.

Note:

• Index validity: The declared input/output index must be within the valid range defined by the model.
• Address consistency: During model execution, ensure that the corresponding input and output are transferred to the same memory address. Otherwise, a runtime error will occur.

Mandatory/Optional: optional

Effective level: graph

ge.exec.outputReuseMemIndexes

Sets whether to enable the memory reuse function for the entire graph output. After the function is enabled, the memory of the entire graph output can be reused as the intermediate memory required during model execution, reducing the memory peak.

If this function is enabled, the value is the index of the entire graph output. If memory reuse is enabled for multiple outputs, use commas (,) to separate multiple indexes.

Note:

• If memory reuse is enabled for the entire graph output, ensure that the output memory size is 32-byte aligned.
• Output indexes are identified based on the output sequence of the entire graph. The index starts from 0.
• If the configured output index is greater than or equal to the value of the output count, index is invalid and the index configuration does not take effect.

Configuration example:

{"ge.exec.outputReuseMemIndexes", "0,1,2"};

Mandatory/Optional: optional

Effective level: graph

ge.externalWeight

When multiple models are loaded in a session, if the weights of these models can be reused, you are advised to use this configuration item to externalize the weights of the Const/Constant nodes on the network to implement weight reuse among multiple models and reduce the memory usage of the weights.

Parameter values:

• 0 (default): The weights are not externalized and are directly saved in the graph.
• 1: The weights are externalized but not normalized. The weight files of all Const/Constant nodes on the network are flushed to disks, and the node type is converted to FileConstant. Weights of different nodes are stored in different files named as weight_<hash>.
• 2: The weights are externalized and normalized. The weight files of all Const/Constant nodes on the network are flushed to disks, and the node type is converted to FileConstant. All weights are saved in the same file named in the format of "original root graph name_weight_combined".

Description of the file flush path:

• If ge.externalWeightDir is configured, the weight files are flushed to the specified directory.
• If the ASCEND_WORK_PATH environment variable is not configured in the environment, the weight files are flushed to the current execution directory tmp_weight_<pid>_<sessionid>.
• If the ASCEND_WORK_PATH environment variable is configured in the environment, the weight files are flushed to the ${ASCEND_WORK_PATH}/tmp_weight_<pid>_<sessionid> directory.

Priority of the flush path: ge.externalWeightDir > ${ASCEND_WORK_PATH}/tmp_weight_<pid>_<sessionid> > current execution directory tmp_weight_<pid>_<sessionid>

When the model is uninstalled, the tmp_weight_<pid>_<sessionid> directory is deleted.

Configuration example:

{"ge.externalWeight", "1"};

Mandatory/Optional: optional

Effective level: global and session

ge.externalWeightDir

Sets the flush path for the external weight file.

Restrictions:

• To specify the flush path for the external weight file, use this parameter together with ge.externalWeight.
• Priority: ge.externalWeightDir > ${ASCEND_WORK_PATH}/tmp_weight_<pid>_<sessionid> > current execution directory tmp_weight_<pid>_<sessionid>

Configuration example:

{"ge.externalWeight", "1"};
{"ge.externalWeightDir", "$HOME/your_tmp_path"};

Mandatory/Optional: optional

Effective level: global and session

ge.inputBatchCpy

Sets whether to enable the batch memory copy function when input data is moved from the host to the device.

The function controlled by this parameter improves the performance of data movement from the host to the device. It applies to the scenario where data needs to be frequently moved and the PCIe bandwidth usage is low. After the function is enabled, bandwidth utilization can be improved.

Parameter values:

• 1: The batch memory copy function is enabled. This value takes effect only when the number of user inputs is greater than 1.
• 0 (default): The batch memory copy function is disabled.

Restrictions:

• This parameter can be used only by the following products:

  Ascend 950PR / Ascend 950DT

  Atlas A3 training products / Atlas A3 inference products

  Atlas A2 training products / Atlas A2 inference products

• The function is usually used in multi-session scenarios. Considering that the number of inputs in different sessions may be different, you are advised to set this parameter at the session level and determine whether to enable this function based on the input. The setting at the global or graph level is not recommended.
• This parameter is passed during session initialization. When the graph is run later, this function can be enabled only by calling the API described in RunGraphAsync.
• If the number of initial network inputs is 1, the batch copy function does not take effect even if it is configured.
• If both ge.exec.input_fusion_size (fusion and copy) and ge.inputBatchCpy (batch copy) are configured, the threshold for the fusion and copy function may affect the batch copy function.

  For example, if a user has five inputs and four of them meet the threshold for the fusion and copy function, the fusion and copy function is performed on the four inputs, and the batch copy function is not performed on the remaining input.

Configuration example:

{"ge.inputBatchCpy", "0"};

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.featureBaseRefreshable

Sets whether the feature memory address can be refreshed. To manage the feature memory and refresh the address for multiple times, set this parameter to the refreshable value.

This parameter applies only to static shape graphs.

Parameter values:

0 (default): The feature memory address cannot be refreshed.

1: The feature memory address of a model can be refreshed.

Configuration example:

{"ge.featureBaseRefreshable", "0"};

Mandatory/Optional: optional

Effective level: global, session, and graph