手上有一台四卡300IDUO推理服务器，操作系统为麒麟v10 arm版本，容器化部署了1.0.0-300I-Duo-py311-openeuler24.03-lts，并按照gitee上的说明文件成功服务化部署deepseek-r1-32B，但按照首页的帖子对mindie集成部署做测试时出现问题。
运行torch_npu.npu.is_available()，返回True
运行a=torch.rand(3,4).npu()，a=a+a,print(a)时，返回如下信息：
[2025-2-23 10:56:12] [ERROR] [ascend310p] 737efd3b48fb8dd663e2cebfdb38ba722ae46b20b4aedccb351d75c8e3615948_te In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/../ascendc/is_finite/is_finite.cpp:22:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/../ascendc/is_finite/is_finite.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/kernel_operator.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tpipe_impl.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/interface/kernel_tpipe.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tpipe_base.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tensor_impl.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/interface/kernel_tensor.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_utils.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/utils/kernel_utils_macros.h:33:
/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_macros.h:24:10: fatal error: 'cstdint' file not found
#include <cstdint>
^~~~~~~~~
1 error generated.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor.py", line 431, in __repr__
return torch._tensor_str._str(self, tensor_contents=tensor_contents)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor_str.py", line 664, in _str
return _str_intern(self, tensor_contents=tensor_contents)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor_str.py", line 595, in _str_intern
tensor_str = _tensor_str(self, indent)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor_str.py", line 347, in _tensor_str
formatter = _Formatter(get_summarized_data(self) if summarize else self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor_str.py", line 138, in __init__
tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:218 OPS function error: IsFinite, error code is 500002
[ERROR] 2025-02-23-18:56:12 (PID:118, Device:0, RankID:-1) ERR01100 OPS call acl api failed
[Error]: A GE error occurs in the system.
Rectify the fault based on the error information in the ascend log.
E40021: [PID: 118] 2025-02-23-18:56:12.834.755 Failed to compile Op [IsFinite1]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/is_finite.py failed with errormsg/stack: File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/tikcpp/compile_op.py", line 530, in dump_build_log
raise Exception("An error occurred during compile phases of {}, msg is {}".\
Exception: An error occurred during compile phases of CompileStage.PRECOMPILE, msg is In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/../ascendc/is_finite/is_finite.cpp:22:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/../ascendc/is_finite/is_finite.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/kernel_operator.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tpipe_impl.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/interface/kernel_tpipe.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tpipe_base.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tensor_impl.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/interface/kernel_tensor.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_utils.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/utils/kernel_utils_macros.h:33:
/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_macros.h:24:10: fatal error: 'cstdint' file not found
#include <cstdint>
^~~~~~~~~
1 error generated.
], optype: [IsFinite])
Solution: See the host log for details, and then check the Python stack where the error log is reported.
TraceBack (most recent call last):
Compile op[IsFinite1] failed, oppath[/usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/is_finite.py], optype[IsFinite], taskID[2]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:367]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[281460967338400] recompile single op[IsFinite1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:1069]
[SubGraphOpt][Compile][ParalCompOp] Thread[281460967338400] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1116]
[SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1190]
[GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1410]
Call OptimizeFusedGraph failed, ret:4294967295, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:119]
subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:815]
build graph failed, graph id:0, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
请问如何解决？
点赞
收藏
回复
分享
举报
已解决
手上有一台四卡300IDUO推理服务器，操作系统为麒麟v10 arm版本，容器化部署了1.0.0-300I-Duo-py311-openeuler24.03-lts，并按照gitee上的说明文件成功服务化部署deepseek-r1-32B，但按照首页的帖子对mindie集成部署做测试时出现问题。
运行torch_npu.npu.is_available()，返回True
运行a=torch.rand(3,4).npu()，a=a+a,print(a)时，返回如下信息：
[2025-2-23 10:56:12] [ERROR] [ascend310p] 737efd3b48fb8dd663e2cebfdb38ba722ae46b20b4aedccb351d75c8e3615948_te In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/../ascendc/is_finite/is_finite.cpp:22:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/../ascendc/is_finite/is_finite.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/kernel_operator.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tpipe_impl.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/interface/kernel_tpipe.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tpipe_base.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tensor_impl.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/interface/kernel_tensor.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_utils.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/utils/kernel_utils_macros.h:33:
/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_macros.h:24:10: fatal error: 'cstdint' file not found
#include <cstdint>
^~~~~~~~~
1 error generated.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor.py", line 431, in __repr__
return torch._tensor_str._str(self, tensor_contents=tensor_contents)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor_str.py", line 664, in _str
return _str_intern(self, tensor_contents=tensor_contents)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor_str.py", line 595, in _str_intern
tensor_str = _tensor_str(self, indent)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor_str.py", line 347, in _tensor_str
formatter = _Formatter(get_summarized_data(self) if summarize else self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor_str.py", line 138, in __init__
tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:218 OPS function error: IsFinite, error code is 500002
[ERROR] 2025-02-23-18:56:12 (PID:118, Device:0, RankID:-1) ERR01100 OPS call acl api failed
[Error]: A GE error occurs in the system.
Rectify the fault based on the error information in the ascend log.
E40021: [PID: 118] 2025-02-23-18:56:12.834.755 Failed to compile Op [IsFinite1]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/is_finite.py failed with errormsg/stack: File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/tikcpp/compile_op.py", line 530, in dump_build_log
raise Exception("An error occurred during compile phases of {}, msg is {}".\
Exception: An error occurred during compile phases of CompileStage.PRECOMPILE, msg is In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/../ascendc/is_finite/is_finite.cpp:22:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/../ascendc/is_finite/is_finite.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/kernel_operator.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tpipe_impl.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/interface/kernel_tpipe.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tpipe_base.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_tensor_impl.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/interface/kernel_tensor.h:24:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_utils.h:23:
In file included from /usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/utils/kernel_utils_macros.h:33:
/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/tikcpp/tikcfw/impl/kernel_macros.h:24:10: fatal error: 'cstdint' file not found
#include <cstdint>
^~~~~~~~~
1 error generated.
], optype: [IsFinite])
Solution: See the host log for details, and then check the Python stack where the error log is reported.
TraceBack (most recent call last):
Compile op[IsFinite1] failed, oppath[/usr/local/Ascend/ascend-toolkit/8.0.0/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/is_finite.py], optype[IsFinite], taskID[2]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:367]
[SubGraphOpt][Compile][ProcFailedCompTask] Thread[281460967338400] recompile single op[IsFinite1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:1069]
[SubGraphOpt][Compile][ParalCompOp] Thread[281460967338400] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1116]
[SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1190]
[GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1410]
Call OptimizeFusedGraph failed, ret:4294967295, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:119]
subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:815]
build graph failed, graph id:0, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
请问如何解决？