Environment Variable List

This document describes the environment variables that can be used when developers build AI applications and services based on CANN.

Environment variables can be implemented using commands, APIs, and configurations, including the export command, putenv/getenv/setenv/unsetenv/clearenv functions, os.environ, and os.getenv. It is recommended that the user should set environment variables before starting application processes. Otherwise, environment variable access conflicts may occur, causing program exceptions.
This document does not describe the environment variables of Ascend Extension for PyTorch. For details about the environment variables of Ascend Extension for PyTorch, see Ascend Extension for PyTorch Environment Variable Reference.

Installation

Environment Variable	Description
ASCEND_CACHE_PATH	Storage path of shared files. Set this environment variable if you require all shared files generated during component build and runtime to be flushed to a unified directory.
ASCEND_WORK_PATH	Storage path for node-exclusive files. Set this environment variable if you require all node-exclusive files generated during component build and runtime to be flushed to a unified directory.
ASCEND_CUSTOM_OPP_PATH	Installation path of the user custom operator package. Set this environment variable if the custom operator package generated during build needs to be installed in a specified path.

Graph Build

Environment Variable	Description
DUMP_GE_GRAPH	Sets the detail level of graph dumps.
DUMP_GRAPH_LEVEL	Specifies the dump level for graph compilation stages.
DUMP_GRAPH_FORMAT	Controls the type of the dump file to be generated.
DUMP_GRAPH_PATH	Sets the path for storing dump graph files. The path can be an absolute path or a relative path of the script execution path.
OP_NO_REUSE_MEM	By default, memory reuse is enabled during graph build on the Ascend platform. In fault locating scenarios, if developers suspect that the computation result is abnormal due to memory reuse errors, they can use this environment variable to allocate memory to an operator separately.
ASCEND_ENGINE_PATH	Set this environment variable when converting a single-operator JSON file to an offline model and you intend to exclusively adopt TBE operators during model conversion. The conversion process will not search for AI CPU operators, and an error will be thrown if the corresponding TBE operator is missing.
MAX_COMPILE_CORE_NUMBER	Specifies the number of CPU cores available for graph build.
MULTI_THREAD_COMPILE	Enables or disables the single-thread build during model conversion.
ENABLE_NETWORK_ANALYSIS_DEBUG	In the TensorFlow training scenario, if computational graph build fails, the training process is terminated by default and the remaining graphs are not delivered to the device. You can set this environment variable to make TF Adapter continuously deliver computational graphs to the device without terminating the training process when graph build fails.

Operator Build

Environment Variable	Description
TE_PARALLEL_COMPILER	Parallel build configuration. Set this environment variable to enable parallel build, which delivers superior performance in large network scenarios.
ASCEND_MAX_OP_CACHE_SIZE	Maximum disk capacity of the cache directory under each AI processor. Set this environment variable when the operator build cache function is enabled.
ASCEND_REMAIN_CACHE_SIZE_RATIO	Specifies the percentage of cache space to be reserved when the operator compilation cache function is enabled. When the occupied compilation cache space reaches ASCEND_MAX_OP_CACHE_SIZE and old kernel files need to be purged, the system will reserve a proportion of the cache space as specified by this variable. The default value is 50, measured in percent.
IGNORE_INFER_ERROR	Specifies whether to skip operator prototype deliverable verification when an operator is inserted to a graph. The deliverables include the implementation of the adaptation functions for operator insertion into the graph, such as shape deduction.

Resource Configuration

Environment Variable	Description
ASCEND_DEVICE_ID	Specifies the logical ID of the AI processor used by the current process.
ASCEND_RT_VISIBLE_DEVICES	Specifies the devices that are visible to the current process. One or more device IDs can be specified at a time. By using this environment variable, you can adjust the devices without modifying the application.
AUTO_USE_UC_MEMORY	Controls whether to allow operators to transfer data without passing through the L2 cache.
RESOURCE_CONFIG_PATH	Specifies the path for storing the heterogeneous resource description file.

Operator Execution

Environment Variable	Description
ACLNN_CACHE_LIMIT	Sets the number of operator information entries cached on the host for an aclnn API. The cached operator information includes the workspace size, operator executor, and tiling details.

Graph Execution

Environment Variable	Description
ENABLE_DYNAMIC_SHAPE_MULTI_STREAM	During graph execution, enabling the multi-stream concurrency function can improve network performance in certain scenarios. Currently, the multi-stream concurrency function is disabled by default. If you want to enable this function in dynamic shape scenario, you can use this environment variable to enable it.
MAX_RUNTIME_CORE_NUMBER	In training and online inference scenarios, this environment variable can be used to enable multi-thread task scheduling of the graph executor (host) for the network in dynamic shape graph mode.

TFAdapter

Environment Variable	Description
JOB_ID	Sets a custom task ID in TensorFlow training and online inference scenarios.
ENABLE_FORCE_V2_CONTROL	In the TensorFlow 1.15 training scenario, if the input has a dynamic shape, upgrade the control flow operators of the V1 version to those of the V2 version to support the dynamic shape function. Only TensorFlow V2 control flow operators (such as If, Case, While, For, and PartitionedCall) support dynamic shapes. TensorFlow V1 control flow operators (such as Switch, Merge, Enter, LoopCond, NextIteration, Exit, and ControlTrigger) corresponding to the tf.case, tf.cond, and tf.while_loop APIs do not support dynamic shapes. If the network has many branch structures, upgrade the control flow operators of the V1 version to those of the V2 version. Otherwise, the flow of data may exceed the limit.
ENABLE_HF32_EXECUTION	Enables or disables the function of automatically replacing the FP32 data type with the HF32 data type for the TensorFlow 1.15 network. In the current version, this environment variable takes effect only for Conv and Matmul operators.
NPU_DEBUG	Enables or disables debug logging of the TF Adapter in the TensorFlow 2.6.5 training and online inference scenario.
NPU_DUMP_GRAPH	Enables or disables graph dump of the TF Adapter in the TensorFlow 2.6.5 training and online inference scenario.
NPU_ENABLE_PERF	Prints graph time consumption of the TF Adapter in TensorFlow 2.6.5 training and online inference scenarios.
NPU_LOOP_SIZE	Sets the number of iterations per loop offloaded to the NPU in the TensorFlow 2.6.5 training and online inference scenarios.
STEP_NOW	In the TensorFlow 1.15 training scenario, if the training acceleration function is enabled through the experimental_accelerate_train_mode or accelerate_train_mode parameter, you can use this environment variable to set the number of execution steps on the NPU.
TOTAL_STEP	In TensorFlow 1.15 training scenarios, if training acceleration is enabled via the experimental_accelerate_train_mode or accelerate_train_mode parameter, this environment variable can be used to set the total number of training steps executed on the NPU.
LOSS_NOW	In the TensorFlow 1.15 training scenario, if the training acceleration function is enabled through the experimental_accelerate_train_mode or accelerate_train_mode parameter, you can use this environment variable to set the loss value of the current iteration on the NPU.
TARGET_LOSS	In the TensorFlow 1.15 training scenario, if the training acceleration function is enabled through the experimental_accelerate_train_mode or accelerate_train_mode parameter, you can use this environment variable to set the target training loss value on the NPU.
RANK_TABLE_FILE	In TensorFlow distributed training or inference scenarios, this environment variable is used to specify the rank table resource configuration file of the AI processor involved in collective communication, including the path and name of the rank table file.
RANK_ID	Sets the rank ID of the current process in the collective communication process group in the TensorFlow distributed training or inference scenario.
RANK_SIZE	Sets the number of devices corresponding to the current training process in TensorFlow distributed training or inference scenarios.
CM_CHIEF_IP	In the TensorFlow distributed training scenario, you can choose not to use the rank table file. Instead, use the environment variables to automatically generate resource information and initialize the collective communication component. This environment variable is used to configure the listening host IP address of the master node.
CM_CHIEF_PORT	Configures the listening port of the master node.
CM_CHIEF_DEVICE	Configures the logical ID of the device for collecting statistics on the server cluster on the master node.
CM_WORKER_SIZE	Configures the number of devices in the service communicator.
CM_WORKER_IP	Configures the NIC IP address used for information exchange between the current device and the master node.

Collective Communication

Environment Variable	Description
Function
HCCL_CONNECT_TIMEOUT	Configures the timeout wait period of socket connection establishment between different devices in the distributed training or inference scenario. The progress of collective communication initialization varies depending on the device. This environment variable synchronizes the progress of socket establishment between devices by using a timeout interval.
HCCL_EXEC_TIMEOUT	During distributed training or inference, tasks executed by different device processes may be inconsistent, for example, only specific processes save the checkpoint data. This environment variable controls the synchronization wait time during task execution between devices. Within this configured time, each device process waits for other devices to perform communication synchronization.
HCCL_ALGO	Configures the communication algorithms between servers and supernodes. The algorithms can be configured globally or by operator.
HCCL_BUFFSIZE	Sets the size of the shared data buffer used by the communicator. The value must be an integer greater than or equal to 1. The default value is 200. The unit is MB.
HCCL_INTRA_PCIE_ENABLE	Specifies whether to use the PCIe link for communication on a server.
HCCL_INTRA_ROCE_ENABLE	Specifies whether to use the RoCE link for communication on a server or supernode.
HCCL_INTER_HCCS_DISABLE	Sets the type of the communication link between supernodes in supernode mode. The options are as follows:
HCCL_OP_EXPANSION_MODE	Configures the expansion mode of communication operators.
HCCL_DETERMINISTIC	Enables or disables the deterministic computation or order-preserving function for reduction communication operators, including AllReduce, ReduceScatter, ReduceScatterV, and Reduce. Order-preserving reduction refers to strict deterministic computation, which guarantees uniform reduction order while maintaining deterministic computation.
HCCL_LOGIC_SUPERPOD_ID	For the supernode mode networking of the Atlas A3 training product / Atlas A3 inference product , if the rank table file is not used to configure cluster resource information, you can use this environment variable to specify the ID of the supernode to which the process running on the current node belongs. In this way, a physical supernode can be divided into multiple logical supernodes.
Performance
HCCL_RDMA_PCIE_DIRECT_POST_NOSTRICT	Submits the RDMA tasks in PCIe Direct mode in multi-server communication scenarios where the host OS uses non-4 KB memory pages and the communication operator delivery performance encounters the host bound. This helps improve the communication operator delivery performance.
HCCL_RDMA_QPS_PER_CONNECTION	Sets the number of queue pairs (QPs) used for data transmission during RDMA communication between two ranks. By default, one QP is created.
HCCL_RDMA_QP_PORT_CONFIG_PATH	By default, one queue pair (QP) is created for data transfer during RDMA communication between two ranks. If you want to use multiple QPs for RDMA communication between two ranks and specify the source port numbers used for multi-QP communication, you can use this environment variable.
HCCL_MULTI_QP_THRESHOLD	Sets the minimum amount of data shared by each QP during RDMA communication between ranks through multi-QPs.
Network
HCCL_IF_IP	Configures the communication IP address used by the host during HCCL initialization when the communicator is created based on root node information. This IP address is used to communicate with the root node to create a communicator.
HCCL_IF_BASE_PORT	Specifies the start port number of the host NIC when the communicator is created based on root node information. After the configuration, the system uses 32 ports starting from this port by default to collect cluster information.
HCCL_HOST_SOCKET_PORT_RANGE	Configures the communication port used by HCCL on the host when the communicator is created based on root node information.
HCCL_NPU_SOCKET_PORT_RANGE	Configures the communication port used by HCCL on the NPU when the communicator is created based on root node information.
HCCL_SOCKET_IFNAME	Sets the name of the NIC used by the host during HCCL initialization. HCCL obtains the host IP address based on the NIC name and communicates with the root node to create a communicator.
HCCL_SOCKET_FAMILY	Sets the IP protocol used by the communication NIC.
HCCL_RDMA_TC	Sets the traffic class of the RDMA NIC.
HCCL_RDMA_SL	Sets the service level (SL) of the RDMA NIC. The value must be the same as the PFC priority set for the NIC. Otherwise, performance may deteriorate.
HCCL_RDMA_TIMEOUT	Configures the retransmission timeout of the RDMA NIC.
HCCL_RDMA_RETRY_CNT	Configures the number of retransmission times of the RDMA NIC. The value must be an integer ranging from 1 to 7. The default value is 7.
Debugging
HCCL_DIAGNOSE_ENABLE	Sets whether to cache detailed information about some tasks during collective communication. If a task fails to be executed, detailed logs can be printed for fault locating.
HCCL_ENTRY_LOG_ENABLE	Controls whether to print the runtime logs of the communication operator in real time.
HCCL_DEBUG_CONFIG	Specifies which HCCL submodules output detailed runtime information to the runtime logs (that is, logs under the $HOME/ascend/log/run directory). Valid values include ALG (or alg) (algorithm orchestration module), TASK (or task) (task orchestration module), and RESOURCE (or resource) (resource management module, covering resource allocation and release operations).
HCCL_DFS_CONFIG	The following options are supported:
Reliability
HCCL_OP_RETRY_ENABLE	Enables or disables the retry feature of the HCCL operator. HCCL operator retry is based on the communicator. If an SDMA or RDMA CQE error is reported during the execution of a communication operator, HCCL attempts to retry the communication operator.
HCCL_OP_RETRY_PARAMS	Enables or disables the retry feature of the HCCL operator. HCCL operator retry is based on the communicator. If an SDMA or RDMA CQE error is reported during the execution of a communication operator, HCCL attempts to retry the communication operator.
Security
HCCL_WHITELIST_DISABLE	Enables or disables the HCCL communication trustlist.
HCCL_WHITELIST_FILE	Configures the path of the HCCL communication trustlist configuration file after the communication trustlist verification function is enabled via HCCL_WHITELIST_DISABLE. Only IP addresses in the communication trustlist can be used for collective communication.

AOE Tuning

Environment Variable	Description
TUNE_BANK_PATH	Sets the path of the custom repository generated after Auto Tune.
REPEAT_TUNE	Initiates tuning again. This environment variable takes effect only when subgraph tuning or operator tuning is enabled.
AOE_MODE	Sets the AOE tuning mode in online inference and training scenarios.

AMCT Model Compression

Environment Variable	Description
AMCT_LOG_FILE_LEVEL	Sets the log level of framework-specific log files (amct_pytorch.log for PyTorch; amct_caffe.log for Caffe; amct_onnx.log for ONNX), as well as the log level of logs output by each quantization layer during precision simulation model generation.
AMCT_LOG_LEVEL	Sets the log level of information printed to the screen. This environment variable applies only to quantization of the PyTorch framework, Caffe framework, and ONNX network model.
DUMP_AMCT_RECORD	Specifies whether to generate quantization factors for weights and data. This environment variable applies only to model compression of the MindSpore framework.
AMCT_LOG_DUMP	Configures log flushing behaviors during post-training quantization. This environment variable applies only to quantization performed by calling aclgrphCalibration.

Profiling

Environment Variable	Description
PROFILING_MODE	Enables or disables the profiling function.
PROFILING_OPTIONS	Sets profiling configuration options in training or online inference scenarios.

Logging

Environment Variable	Description
ASCEND_PROCESS_LOG_PATH	Sets the log flush path.
ASCEND_SLOG_PRINT_TO_STDOUT	Enables or disables log printing. After this function is enabled, logs are not saved in the log file. Instead, the generated logs are directly printed and displayed.
ASCEND_GLOBAL_LOG_LEVEL	Sets the level of application logs and module logs. Only debug logs are supported.
ASCEND_MODULE_LOG_LEVEL	Sets the level of each module of app logs. Only debug logs are supported.
ASCEND_GLOBAL_EVENT_ENABLE	Enables or disables event logging for applications.
ASCEND_LOG_DEVICE_FLUSH_TIMEOUT	Specifies the timeout for flushing app logs from the device to the host.
ASCEND_HOST_LOG_FILE_NUM	Sets the number of log files of each process stored in the application log directories (plog and device-id) in the Ascend EP scenario.
ASCEND_COREDUMP_SIGNAL	Sets the core dump semaphore for trace processing.
ASCEND_LOG_SYNC_SAVE	Specifies the processing mode for log congestion.
ASCEND_TRACE_RECORD_NUM	Sets the aging policy for trace log files. The value range is [10, 1000].

Fault Information Collection

Environment Variable	Description
NPU_COLLECT_PATH	Specifies the storage path for fault information, including dump graphs, abnormal data of AI Core operators, and operator compilation logs. The path can be an absolute path or a relative path (relative to the working directory of the running program or command), on which users must have the read, write, and execute permissions. If the target path does not exist, the system will automatically create all missing directories under the specified path.
ASCEND_DUMP_SCENE	Enables or disables abnormal operator dump when reproducing fault scenarios. When this function is enabled, input and output data, workspace details, and tiling information of faulty operators are exported.
ASCEND_DUMP_PATH	Specifies the storage directory for abnormal operator dump data. Both absolute paths and relative paths (relative to the executable program) are supported. The specified path can contain uppercase letters, lowercase letters, digits, underscores (_), hyphens (-), and periods (.). The user must have read, write, and execute permissions on the path. If the target directory does not exist, the system will automatically create all missing directories along the path. If this path is not configured, abnormal operator dump data will be saved in the current execution directory of the application by default.

Environment Variables to Be Deprecated in Later Versions

Environment Variable	Description
GE_USE_STATIC_MEMORY	Configures the memory allocation mode used during network running.
ENABLE_ACLNN	Determines whether to call the host execution function registered by the operator during graph execution to implement the host execution logic and kernel delivery.