[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:33.669.053 [mindspore/run_check/_check_version.py:388] CheckFailed: name 'sys_version' is not defined
[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:33.669.189 [mindspore/run_check/_check_version.py:389] MindSpore relies on whl packages of "te" and "hccl" in the "latest" folder of the Ascend AI software package (Ascend Data Center Solution). Please check whether they are installed correctly or not, refer to the match info on: https://www.mindspore.cn/install
[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:35.357.284 [mindspore/run_check/_check_version.py:388] CheckFailed: cannot import name 'utils' from partially initialized module 'tbe.common' (most likely due to a circular import) (/root/anaconda3/envs/py39/lib/python3.9/site-packages/tbe/common/__init__.py)
[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:35.357.406 [mindspore/run_check/_check_version.py:389] MindSpore relies on whl packages of "te" and "hccl" in the "latest" folder of the Ascend AI software package (Ascend Data Center Solution). Please check whether they are installed correctly or not, refer to the match info on: https://www.mindspore.cn/install
[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:35.361.757 [mindspore/run_check/_check_version.py:388] CheckFailed: cannot import name 'utils' from partially initialized module 'tbe.common' (most likely due to a circular import) (/root/anaconda3/envs/py39/lib/python3.9/site-packages/tbe/common/__init__.py)
[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:35.361.837 [mindspore/run_check/_check_version.py:389] MindSpore relies on whl packages of "te" and "hccl" in the "latest" folder of the Ascend AI software package (Ascend Data Center Solution). Please check whether they are installed correctly or not, refer to the match info on: https://www.mindspore.cn/install
2024-09-04 09:32:35,527 - mindformers[mindformers/models/auto/configuration_auto.py:318] - INFO - the content in /root/llmDownloads/moxing/predict_glm3_6b.yaml is used for config building.
2024-09-04 09:32:35,529 - mindformers[mindformers/version_control.py:61] - INFO - The Cell Reuse compilation acceleration feature is not supported when the environment variable ENABLE_CELL_REUSE is 0 or MindSpore version is earlier than 2.1.0 or stand_alone mode or pipeline_stages <= 1
2024-09-04 09:32:35,529 - mindformers[mindformers/version_control.py:65] - INFO -
The current ENABLE_CELL_REUSE=0, please set the environment variable as follows:
export ENABLE_CELL_REUSE=1 to enable the Cell Reuse compilation acceleration feature.
2024-09-04 09:32:35,529 - mindformers[mindformers/version_control.py:71] - INFO - The Cell Reuse compilation acceleration feature does not support single-card mode.This feature is disabled by default. ENABLE_CELL_REUSE=1 does not take effect.
2024-09-04 09:32:35,529 - mindformers[mindformers/version_control.py:74] - INFO - The Cell Reuse compilation acceleration feature only works in pipeline parallel mode(pipeline_stage>1).Current pipeline stage=1, the feature is disabled by default.
Traceback (most recent call last):
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/tools/register/register.py", line 192, in get_instance_from_cfg
return obj_cls(**args)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/version_control.py", line 77, in decorator
func(*args, **kwargs)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/glm2/glm2.py", line 168, in __init__
super(ChatGLM2ForConditionalGeneration, self).__init__(config, **kwargs)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/modeling_utils.py", line 254, in __init__
super().__init__(**kwargs)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindspore/nn/cell.py", line 140, in __init__
init_pipeline()
RuntimeError: Initialize GE failed!
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EC0010: 2024-09-04-09:32:36.570.715 Failed to import Python module [ImportError: cannot import name 'utils' from partially initialized module 'tbe.common' (most likely due to a circular import) (/root/anaconda3/envs/py39/lib/python3.9/site-packages/tbe/common/__init__.py).].
Solution: Check that all required components are properly installed and the specified Python path matches the Python installation directory. (If the path does not match the directory, run set_env.sh in the installation package.)
TraceBack (most recent call last):
AOE Failed to call InitCannKB
[GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1719]
[SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79]
[SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120]
[FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:117]
PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:82]
OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:234]
GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:162]
GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:309]
(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:253 InitGe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/llmDownloads/moxing/test.py", line 13, in <module>
model = AutoModel.from_config(config) # 从自定义配置项中实例化模型
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/auto/auto_factory.py", line 298, in from_config
return cls.from_config_origin_mode(config, **kwargs)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/auto/auto_factory.py", line 364, in from_config_origin_mode
model = build_network(config_args.model)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/build_model.py", line 76, in build_network
network = build_model(config, default_args=default_args)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/build_model.py", line 66, in build_model
return MindFormerRegister.get_instance_from_cfg(
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/tools/register/register.py", line 194, in get_instance_from_cfg
raise type(e)('{}: {}'.format(obj_cls.__name__, e))
RuntimeError: ChatGLM2ForConditionalGeneration: Initialize GE failed!
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EC0010: 2024-09-04-09:32:36.570.715 Failed to import Python module [ImportError: cannot import name 'utils' from partially initialized module 'tbe.common' (most likely due to a circular import) (/root/anaconda3/envs/py39/lib/python3.9/site-packages/tbe/common/__init__.py).].
Solution: Check that all required components are properly installed and the specified Python path matches the Python installation directory. (If the path does not match the directory, run set_env.sh in the installation package.)
TraceBack (most recent call last):
AOE Failed to call InitCannKB
[GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1719]
[SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79]
[SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120]
[FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:117]
PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:82]
OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:234]
GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:162]
GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:309]
(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:253 InitGe
点赞
收藏
回复
分享
举报
已解决
[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:33.669.053 [mindspore/run_check/_check_version.py:388] CheckFailed: name 'sys_version' is not defined
[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:33.669.189 [mindspore/run_check/_check_version.py:389] MindSpore relies on whl packages of "te" and "hccl" in the "latest" folder of the Ascend AI software package (Ascend Data Center Solution). Please check whether they are installed correctly or not, refer to the match info on: https://www.mindspore.cn/install
[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:35.357.284 [mindspore/run_check/_check_version.py:388] CheckFailed: cannot import name 'utils' from partially initialized module 'tbe.common' (most likely due to a circular import) (/root/anaconda3/envs/py39/lib/python3.9/site-packages/tbe/common/__init__.py)
[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:35.357.406 [mindspore/run_check/_check_version.py:389] MindSpore relies on whl packages of "te" and "hccl" in the "latest" folder of the Ascend AI software package (Ascend Data Center Solution). Please check whether they are installed correctly or not, refer to the match info on: https://www.mindspore.cn/install
[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:35.361.757 [mindspore/run_check/_check_version.py:388] CheckFailed: cannot import name 'utils' from partially initialized module 'tbe.common' (most likely due to a circular import) (/root/anaconda3/envs/py39/lib/python3.9/site-packages/tbe/common/__init__.py)
[ERROR] ME(52223:140418837182272,MainProcess):2024-09-04-09:32:35.361.837 [mindspore/run_check/_check_version.py:389] MindSpore relies on whl packages of "te" and "hccl" in the "latest" folder of the Ascend AI software package (Ascend Data Center Solution). Please check whether they are installed correctly or not, refer to the match info on: https://www.mindspore.cn/install
2024-09-04 09:32:35,527 - mindformers[mindformers/models/auto/configuration_auto.py:318] - INFO - the content in /root/llmDownloads/moxing/predict_glm3_6b.yaml is used for config building.
2024-09-04 09:32:35,529 - mindformers[mindformers/version_control.py:61] - INFO - The Cell Reuse compilation acceleration feature is not supported when the environment variable ENABLE_CELL_REUSE is 0 or MindSpore version is earlier than 2.1.0 or stand_alone mode or pipeline_stages <= 1
2024-09-04 09:32:35,529 - mindformers[mindformers/version_control.py:65] - INFO -
The current ENABLE_CELL_REUSE=0, please set the environment variable as follows:
export ENABLE_CELL_REUSE=1 to enable the Cell Reuse compilation acceleration feature.
2024-09-04 09:32:35,529 - mindformers[mindformers/version_control.py:71] - INFO - The Cell Reuse compilation acceleration feature does not support single-card mode.This feature is disabled by default. ENABLE_CELL_REUSE=1 does not take effect.
2024-09-04 09:32:35,529 - mindformers[mindformers/version_control.py:74] - INFO - The Cell Reuse compilation acceleration feature only works in pipeline parallel mode(pipeline_stage>1).Current pipeline stage=1, the feature is disabled by default.
Traceback (most recent call last):
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/tools/register/register.py", line 192, in get_instance_from_cfg
return obj_cls(**args)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/version_control.py", line 77, in decorator
func(*args, **kwargs)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/glm2/glm2.py", line 168, in __init__
super(ChatGLM2ForConditionalGeneration, self).__init__(config, **kwargs)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/modeling_utils.py", line 254, in __init__
super().__init__(**kwargs)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindspore/nn/cell.py", line 140, in __init__
init_pipeline()
RuntimeError: Initialize GE failed!
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EC0010: 2024-09-04-09:32:36.570.715 Failed to import Python module [ImportError: cannot import name 'utils' from partially initialized module 'tbe.common' (most likely due to a circular import) (/root/anaconda3/envs/py39/lib/python3.9/site-packages/tbe/common/__init__.py).].
Solution: Check that all required components are properly installed and the specified Python path matches the Python installation directory. (If the path does not match the directory, run set_env.sh in the installation package.)
TraceBack (most recent call last):
AOE Failed to call InitCannKB
[GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1719]
[SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79]
[SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120]
[FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:117]
PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:82]
OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:234]
GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:162]
GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:309]
(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:253 InitGe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/llmDownloads/moxing/test.py", line 13, in <module>
model = AutoModel.from_config(config) # 从自定义配置项中实例化模型
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/auto/auto_factory.py", line 298, in from_config
return cls.from_config_origin_mode(config, **kwargs)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/auto/auto_factory.py", line 364, in from_config_origin_mode
model = build_network(config_args.model)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/build_model.py", line 76, in build_network
network = build_model(config, default_args=default_args)
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/models/build_model.py", line 66, in build_model
return MindFormerRegister.get_instance_from_cfg(
File "/root/anaconda3/envs/py39/lib/python3.9/site-packages/mindformers/tools/register/register.py", line 194, in get_instance_from_cfg
raise type(e)('{}: {}'.format(obj_cls.__name__, e))
RuntimeError: ChatGLM2ForConditionalGeneration: Initialize GE failed!
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EC0010: 2024-09-04-09:32:36.570.715 Failed to import Python module [ImportError: cannot import name 'utils' from partially initialized module 'tbe.common' (most likely due to a circular import) (/root/anaconda3/envs/py39/lib/python3.9/site-packages/tbe/common/__init__.py).].
Solution: Check that all required components are properly installed and the specified Python path matches the Python installation directory. (If the path does not match the directory, run set_env.sh in the installation package.)
TraceBack (most recent call last):
AOE Failed to call InitCannKB
[GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1719]
[SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79]
[SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120]
[FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:117]
PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:82]
OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:234]
GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:162]
GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:309]
(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:253 InitGe