ASCEND_DUMP_SCENE

Description

Enables or disables abnormal operator dump when reproducing fault scenarios. When this function is enabled, input and output data, workspace details, and tiling information of faulty operators are exported.

The following options are supported:

• aic_err_brief_dump: lightweight exception dump, which exports the input, output, and workspace data of abnormal operators of the AI Core.
• aic_err_norm_dump: common exception dump, which exports the shape, data type, format, and attribute information in addition to the lightweight exception dump.
• aic_err_detail_dump: exports the internal storage, register, and call stack information of the AI Core in addition to the lightweight exception dump.

  When configuring this option, note that:

  • This option is supported only by the following models:

    Atlas A2 training product/Atlas A2 inference product

    Atlas A3 training product/Atlas A3 inference product

  • During dump file export, the AI Core running the faulty operator will be suspended, which may disrupt normal execution of other service processes on the device. The AI Core will automatically resume operation after dump files are exported.
  • Once dump files are exported, the host-side user service process will be forcibly terminated. Any error logs generated during this forced termination shall not be used for AI Core fault analysis.
  • If aic_err_detail_dump is configured and dump files are generated without any *.core files, it indicates that aic_err_detail_dump failed to take effect. The system will automatically fall back to the aic_err_brief_dump mode.

Example

export ASCEND_DUMP_SCENE=aic_err_brief_dump

Constraints

• There are multiple ways to enable the abnormal operator dump function, sorted by priority from highest to lowest: ASCEND_DUMP_SCENE environment variable > NPU_COLLECT_PATH environment variable > configuration files (via aclInit and aclmdlSetDump APIs).
• For dump data storage paths exported using this environment variable, the priority order is as follows: ASCEND_DUMP_PATH environment variable > ASCEND_WORK_PATH environment variable > Default path (the application's working directory).

Applicability

Atlas 350 Accelerator Card

Atlas A3 training product/Atlas A3 inference product

Atlas A2 training product/Atlas A2 inference product

Atlas 200I/500 A2 inference product

Atlas inference product

Atlas training product

MC62CM12A AI Processor