使用Atls 800T A2训练ChatGLM 6B-PyTorch（https://www.hiascend.com/software/modelzoo/models/detail/892001d920694836ae7b7c25c3b5a066）在运行bash preprocess.sh（预处理数据集）时报错RuntimeError: Initialize:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:217 NPU function error: at_npu::native::AclSetCompileopt(aclCompileOpt::ACL_PRECISION_MODE, precision_mode), error code is 500001
安装的npu驱动和固件版本为1.0.25.alpha，cann的版本为8.0.RC3.alpha003。（https://www.hiascend.com/hardware/firmware-drivers/community?product=4&model=26&cann=8.0.RC3.alpha003&driver=1.0.25.alpha）
然后后续在安装pytorch框架和torch_npu时发现，适配的cann版本似乎只有8.0.RC2，不知道是否是这个版本不匹配的原因导致的报错。（https://www.hiascend.com/document/detail/zh/Pytorch/60RC2/configandinstg/instg/insg_0005.html）
我另外也有跑AscendC算子开发的快速入门，基于同一个cann环境，是可以跑成功的。
具体报错如下：
2024-09-30 11:17:18.981 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=0.
2024-09-30 11:17:19.106 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=1.
2024-09-30 11:17:19.229 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=2.
2024-09-30 11:17:19.353 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=3.
2024-09-30 11:17:19.477 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=4.
2024-09-30 11:17:19.601 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=5.
2024-09-30 11:17:19.725 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=6.
2024-09-30 11:17:19.849 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=7.
2024-09-30 11:17:19.856 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(2)[DISABLE], device id=0.
/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/torch/lib:/usr/local/lib:/usr/local/lib64/python3.9/site-packages/torch/lib:/usr/local/lib:/usr/local/lib/python3.9/site-packages/torch/lib:/usr/lib:/usr/lib64/python3.9/site-packages/torch/lib:/usr/lib:/usr/lib/python3.9/site-packages/torch/lib:
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.
Traceback (most recent call last): File "/data_home/cly/ModelZoo-PyTorch/PyTorch/built-in/foundation/ChatGLM-6B/ptuning/preprocess.py", line 45, in <module>
import torch
File "/usr/local/lib64/python3.9/site-packages/torch/__init__.py", line 1382, in <module>
from .functional import * # noqa: F403
File "/usr/local/lib64/python3.9/site-packages/torch/functional.py", line 7, in <module>
import torch.nn.functional as F
File "/usr/local/lib64/python3.9/site-packages/torch/nn/__init__.py", line 1, in <module>
from .modules import * # noqa: F403
File "/usr/local/lib64/python3.9/site-packages/torch/nn/modules/__init__.py", line 35, in <module>
from .transformer import TransformerEncoder, TransformerDecoder, \
File "/usr/local/lib64/python3.9/site-packages/torch/nn/modules/transformer.py", line 20, in <module>
device: torch.device = torch.device(torch._C._get_default_device()), # torch.device('cpu'),
/usr/local/lib64/python3.9/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: _ARRAY_API not found (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
device: torch.device = torch.device(torch._C._get_default_device()), # torch.device('cpu'),
/usr/local/lib64/python3.9/site-packages/torch_npu/contrib/transfer_to_npu.py:301: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************
warnings.warn(msg, ImportWarning)
/usr/local/lib64/python3.9/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
Traceback (most recent call last):
File "/data_home/cly/ModelZoo-PyTorch/PyTorch/built-in/foundation/ChatGLM-6B/ptuning/preprocess.py", line 48, in <module>
import deepspeed_npu
File "/usr/local/lib/python3.9/site-packages/deepspeed_npu/__init__.py", line 6, in <module>
FLAG_SUPPORT_INF_NAN = hasattr(torch_npu.npu.utils, 'is_support_inf_nan') and torch_npu.npu.utils.is_support_inf_nan()
File "/usr/local/lib64/python3.9/site-packages/torch_npu/npu/utils.py", line 333, in is_support_inf_nan
torch_npu.npu._lazy_init()
File "/usr/local/lib64/python3.9/site-packages/torch_npu/npu/__init__.py", line 214, in _lazy_init
torch_npu._C._npu_init()
RuntimeError: Initialize:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:217 NPU function error: at_npu::native::AclSetCompileopt(aclCompileOpt::ACL_PRECISION_MODE, precision_mode), error code is 500001
[ERROR] 2024-09-30-11:17:27 (PID:844217, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The internal ACL of the system is incorrect.
Rectify the fault based on the error information in the ascend log.
EC0010: [PID: 844217] 2024-09-30-11:17:27.083.944 Failed to import Python module [AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead..].
Solution: Check that all required components are properly installed and the specified Python path matches the Python installation directory. (If the path does not match the directory, run set_env.sh in the installation package.)
TraceBack (most recent call last):
AOE Failed to call InitCannKB[FUNC:Initialize][FILE:python_adapter_manager.cc][LINE:47]
Failed to initialize TeConfigInfo.
[GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1721]
[SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79]
[SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120]
[FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:117]
PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:82]
OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:234]
GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:162]
GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:334]
[Initialize][Ge]GEInitialize failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Init][Compiler]Init compiler failed[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
[Set][Options]OpCompileProcessor init failed![FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
/usr/lib64/python3.9/tempfile.py:830: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpih9yzdyi'>
_warnings.warn(warn_message, ResourceWarning)
点赞
收藏
回复
分享
举报
已解决
使用Atls 800T A2训练ChatGLM 6B-PyTorch（https://www.hiascend.com/software/modelzoo/models/detail/892001d920694836ae7b7c25c3b5a066）在运行bash preprocess.sh（预处理数据集）时报错RuntimeError: Initialize:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:217 NPU function error: at_npu::native::AclSetCompileopt(aclCompileOpt::ACL_PRECISION_MODE, precision_mode), error code is 500001
安装的npu驱动和固件版本为1.0.25.alpha，cann的版本为8.0.RC3.alpha003。（https://www.hiascend.com/hardware/firmware-drivers/community?product=4&model=26&cann=8.0.RC3.alpha003&driver=1.0.25.alpha）
然后后续在安装pytorch框架和torch_npu时发现，适配的cann版本似乎只有8.0.RC2，不知道是否是这个版本不匹配的原因导致的报错。（https://www.hiascend.com/document/detail/zh/Pytorch/60RC2/configandinstg/instg/insg_0005.html）
我另外也有跑AscendC算子开发的快速入门，基于同一个cann环境，是可以跑成功的。
具体报错如下：
2024-09-30 11:17:18.981 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=0.
2024-09-30 11:17:19.106 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=1.
2024-09-30 11:17:19.229 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=2.
2024-09-30 11:17:19.353 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=3.
2024-09-30 11:17:19.477 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=4.
2024-09-30 11:17:19.601 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=5.
2024-09-30 11:17:19.725 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=6.
2024-09-30 11:17:19.849 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=7.
2024-09-30 11:17:19.856 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(2)[DISABLE], device id=0.
/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/torch/lib:/usr/local/lib:/usr/local/lib64/python3.9/site-packages/torch/lib:/usr/local/lib:/usr/local/lib/python3.9/site-packages/torch/lib:/usr/lib:/usr/lib64/python3.9/site-packages/torch/lib:/usr/lib:/usr/lib/python3.9/site-packages/torch/lib:
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.
Traceback (most recent call last): File "/data_home/cly/ModelZoo-PyTorch/PyTorch/built-in/foundation/ChatGLM-6B/ptuning/preprocess.py", line 45, in <module>
import torch
File "/usr/local/lib64/python3.9/site-packages/torch/__init__.py", line 1382, in <module>
from .functional import * # noqa: F403
File "/usr/local/lib64/python3.9/site-packages/torch/functional.py", line 7, in <module>
import torch.nn.functional as F
File "/usr/local/lib64/python3.9/site-packages/torch/nn/__init__.py", line 1, in <module>
from .modules import * # noqa: F403
File "/usr/local/lib64/python3.9/site-packages/torch/nn/modules/__init__.py", line 35, in <module>
from .transformer import TransformerEncoder, TransformerDecoder, \
File "/usr/local/lib64/python3.9/site-packages/torch/nn/modules/transformer.py", line 20, in <module>
device: torch.device = torch.device(torch._C._get_default_device()), # torch.device('cpu'),
/usr/local/lib64/python3.9/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: _ARRAY_API not found (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
device: torch.device = torch.device(torch._C._get_default_device()), # torch.device('cpu'),
/usr/local/lib64/python3.9/site-packages/torch_npu/contrib/transfer_to_npu.py:301: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************
warnings.warn(msg, ImportWarning)
/usr/local/lib64/python3.9/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
Traceback (most recent call last):
File "/data_home/cly/ModelZoo-PyTorch/PyTorch/built-in/foundation/ChatGLM-6B/ptuning/preprocess.py", line 48, in <module>
import deepspeed_npu
File "/usr/local/lib/python3.9/site-packages/deepspeed_npu/__init__.py", line 6, in <module>
FLAG_SUPPORT_INF_NAN = hasattr(torch_npu.npu.utils, 'is_support_inf_nan') and torch_npu.npu.utils.is_support_inf_nan()
File "/usr/local/lib64/python3.9/site-packages/torch_npu/npu/utils.py", line 333, in is_support_inf_nan
torch_npu.npu._lazy_init()
File "/usr/local/lib64/python3.9/site-packages/torch_npu/npu/__init__.py", line 214, in _lazy_init
torch_npu._C._npu_init()
RuntimeError: Initialize:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:217 NPU function error: at_npu::native::AclSetCompileopt(aclCompileOpt::ACL_PRECISION_MODE, precision_mode), error code is 500001
[ERROR] 2024-09-30-11:17:27 (PID:844217, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The internal ACL of the system is incorrect.
Rectify the fault based on the error information in the ascend log.
EC0010: [PID: 844217] 2024-09-30-11:17:27.083.944 Failed to import Python module [AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead..].
Solution: Check that all required components are properly installed and the specified Python path matches the Python installation directory. (If the path does not match the directory, run set_env.sh in the installation package.)
TraceBack (most recent call last):
AOE Failed to call InitCannKB[FUNC:Initialize][FILE:python_adapter_manager.cc][LINE:47]
Failed to initialize TeConfigInfo.
[GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1721]
[SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79]
[SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120]
[FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:117]
PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:82]
OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:234]
GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:162]
GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:334]
[Initialize][Ge]GEInitialize failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Init][Compiler]Init compiler failed[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
[Set][Options]OpCompileProcessor init failed![FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
/usr/lib64/python3.9/tempfile.py:830: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpih9yzdyi'>
_warnings.warn(warn_message, ResourceWarning)