NPU_COLLECT_PATH
Description
Specifies the storage path for fault information, including dump graphs, abnormal data of AI Core operators, and operator compilation logs. The path can be an absolute path or a relative path (relative to the working directory of the running program or command), on which users must have the read, write, and execute permissions. If the target path does not exist, the system will automatically create all missing directories under the specified path.
Notes for using this environment variable:
- When this environment variable is set, the dump data collection of abnormal operators is automatically enabled by default. If the operator input or output contains sensitive user information, there may be risks of information leakage.
- When this environment variable is configured and model conversion is performed, extra debug information will be embedded during OM model compilation, which increases the OM file size. If you need to optimize memory layout or have limited memory resources, unset this environment variable once debugging finishes.
- With this variable enabled, only L1 exception dump data will be collected. Model dump, single-operator dump, overflow operator dump, and L0 exception dump data will not be collected.
The priority of the directory for storing L1 exception dump data is as follows: NPU_COLLECT_PATH -> ASCEND_WORK_PATH -> Default path (the extra-info subdirectory under the program's working directory).
L1 exception dump refers to standard exception dump, while L0 exception dump is lightweight exception dump. Both export information including operator input/output and workspace data. However, L1 exception dump outputs more complete data. When L1 exception dump is enabled, the dtype of each tensor, operator names, and associated kernels will be printed to host application logs (plogs).
- When this environment variable is set, operators will be compiled online during model compilation instead of reusing precompiled operator binaries.
The directory for storing the operator information (.o and .json files) compiled online has the following priority: NPU_COLLECT_PATH -> ASCEND_CACHE_PATH -> Default path (${HOME}/atc_data).
Example
export NPU_COLLECT_PATH=$HOME/demo/
Constraints
- When single-operator APIs (for example, aclnn APIs) are called, precompiled operator binaries are used, and online operator compilation is not triggered.
- If the environment variable NPU_COLLECT_PATH is set, the operator compilation files (including .o and .json files) for the following operators in graph mode cannot be generated in the path specified by this environment variable:
MatMulAllReduce
MatMulAllReduceAddRmsNorm
AllGatherMatMul
MatMulReduceScatter
AlltoAllAllGatherBatchMatMul
BatchMatMulReduceScatterAlltoAll
- If the environment variable NPU_COLLECT_PATH is configured, the function of global memory out-of-bounds detection cannot be enabled. Otherwise, errors will be reported when the compiled model file or operator kernel package is used. Global memory out-of-bounds detection can be enabled via the following methods:
- ATC model conversion tool: Add the oom configuration in the file specified by the --op_debug_config parameter. For details, see ATC.
- op_compiler operator compilation tool: Add the oom configuration in the file specified by the --op_debug_config parameter. For details, see Operator Compilation Tool.
- Ascend Graph: Add the oom configuration in the file specified by the op_debug_config or OP_DEBUG_CONFIG parameter. For details, see Graph Development.
- Training scripts developed based on TensorFlow Python APIs for training on the AI processor: Add the oom configuration in the file specified by the op_debug_config parameter. For details, see TensorFlow 1.15 Model Porting Guide and TensorFlow 2.6.5 Model Porting Guide.
Applicability
Atlas 350 Accelerator Card