一、问题现象（附报错日志上下文）：
2025-7-30日的更新17a219f 在300IDUO机器上依然无法使用，报错：
Traceback (most recent call last):
File "/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_utils/error_code.py", line 43, in wapper
return func(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/dynamo/torchair/core/_backend.py", line 123, in compile
return super(TorchNpuGraph, self).compile()
RuntimeError: EZ9999: Inner Error!
EZ9999: [PID: 489196] 2025-08-07-13:29:21.063.127 numHeads:14 of key must be equal to numHeads:2 of kv when 310P.[FUNC:CheckInputFormatAndLimits][FILE:incre_flash_attention_tiling_check.cc][LINE:320]
TraceBack (most recent call last):
Assert (((funcs->tiling)(reinterpret_cast<gert::TilingContext *>(tiling_context_holder.context_))) == ge::GRAPH_SUCCESS) failed[FUNC:RtParseAndTiling][FILE:op_tiling_rt2.cc][LINE:526]
[GenTask][CalcExtOpRunningParam] CalcTilingSinkRunningParam failed.[FUNC:CalcExtOpRunningParam][FILE:aicore_ops_kernel_builder.cc][LINE:259]
[GenTask][CalcOpRunningParam] CalcExtOpRunningParam failed.[FUNC:CalcOpRunningParam][FILE:aicore_ops_kernel_builder.cc][LINE:227]
Call Calculate op:IncreFlashAttention(IncreFlashAttention) running param failed[FUNC:CalcOpParam][FILE:graph_builder.cc][LINE:211]
[Call][PreRun] Failed, graph_id:1, session_id:1.[FUNC:CompileGraph][FILE:graph_manager.cc][LINE:4545]
[Compile][Graph]Compile graph failed, error code:1343225857, session_id:1, graph_id:1.[FUNC:CompileGraph][FILE:ge_api.cc][LINE:1280]
[ERROR] 2025-08-07-13:29:21 (PID:489196, Device:0, RankID:-1) ERR03005 GRAPH internal error
二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x): 8.1.rc1
--Tensorflow/Pytorch/MindSpore 版本: Pytorch: 2.4.0
--Python 版本 (e.g., Python 3.7.5): 3.10.16
--操作系统版本： openEuler release 24.03 (LTS)
三、测试步：
按照 https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/ACL_PyTorch/built-in/audio/CosyVoice/CosyVoice2 的提示进行，其中在运行atc --framework=5 --soc_version=Ascend310P3 --model ./CosyVoice/CosyVoice2-0.5B/speech_token_md.onnx --output ./CosyVoice/CosyVoice2-0.5B/speech --input_shape="feats:1,128,-1;feats_length:1" --precision_mode allow_fp32_to_fp16命令时，有个错误如下，但最后om模型还是转出来了（不知道这个错误对最后有没有影响）：
ATC start working now, please wait for a moment.
Warning: set_tiling_params does not take effect because get_op_mode is dynamic
Warning: set_tiling_params does not take effect because get_op_mode is dynamic
..Warning: set_tiling_params does not take effect because get_op_mode is dynamic
.Warning: set_tiling_params does not take effect because get_op_mode is dynamic
....Warning: set_tiling_params does not take effect because get_op_mode is dynamic
Warning: set_tiling_params does not take effect because get_op_mode is dynamic
....Warning: set_tiling_params does not take effect because get_op_mode is dynamic
....Warning: set_tiling_params does not take effect because get_op_mode is dynamic
..Warning: set_tiling_params does not take effect because get_op_mode is dynamic
..Warning: set_tiling_params does not take effect because get_op_mode is dynamic
...................................................................................................................run stack spill compile error: error: stack frame size (16560) exceeds limit (16000) in function 'te_fused_op_round_sub_add_real_div_mul_add_ad2f18287d99c411b9ad5c422da4ea612176ed93c56480b7c6dfb5cef696cda5_299900002_mix_aiv'
1 error generated.
ATC run success, welcome to the next use.
四、日志信息:
(py3_10_5_21) [root@kylinv241 CosyVoice]# python3 infer.py --model_path=./CosyVoice2-0.5B --stream_out
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:291: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.set_default_device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************
warnings.warn(msg, ImportWarning)
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:246: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/dynamo/torchair/__init__.py:3: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/pkg_resources/__init__.py:3147: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
2025-08-07 13:25:14,582 - modelscope - INFO - PyTorch version 2.4.0 Found.
2025-08-07 13:25:14,584 - modelscope - INFO - Loading ast index from /root/.cache/modelscope/ast_indexer
2025-08-07 13:25:14,640 - modelscope - INFO - Loading done! Current index file version is 1.15.0, with md5 46852a64d9493b37403fd080e09efd06 and a total number of 980 components indexed
failed to import ttsfrd, use WeTextProcessing instead
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/lightning/fabric/__init__.py:41: Deprecated call to `pkg_resources.declare_namespace('lightning.fabric')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/pkg_resources/__init__.py:2559: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('lightning')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(parent)
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/lightning/pytorch/__init__.py:37: Deprecated call to `pkg_resources.declare_namespace('lightning.pytorch')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/pkg_resources/__init__.py:2559: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('lightning')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(parent)
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/diffusers/models/lora.py:393: FutureWarning: `LoRACompatibleLinear` is deprecated and will be removed in version 1.0.0. Use of `LoRACompatibleLinear` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.
deprecate("LoRACompatibleLinear", "1.0.0", deprecation_message)
[INFO] input frame rate=25
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:65: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
warnings.warn(
2025-08-07 13:25:40,980 WETEXT INFO building fst for zh_normalizer ...
[INFO] building fst for zh_normalizer ...
2025-08-07 13:26:22,627 WETEXT INFO done
[INFO] done
2025-08-07 13:26:22,627 WETEXT INFO fst path: /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/zh_tn_tagger.fst
[INFO] fst path: /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/zh_tn_tagger.fst
2025-08-07 13:26:22,627 WETEXT INFO /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/zh_tn_verbalizer.fst
[INFO] /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/zh_tn_verbalizer.fst
2025-08-07 13:26:22,635 WETEXT INFO found existing fst: /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/en_tn_tagger.fst
[INFO] found existing fst: /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/en_tn_tagger.fst
2025-08-07 13:26:22,635 WETEXT INFO /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/en_tn_verbalizer.fst
[INFO] /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/en_tn_verbalizer.fst
2025-08-07 13:26:22,635 WETEXT INFO skip building fst for en_normalizer ...
[INFO] skip building fst for en_normalizer ...
[WARN] acl repeat initialize
[INFO] acl init success
[INFO] open device 0 success
[INFO] get current context
[INFO] load model ./CosyVoice2-0.5B/flow_linux_aarch64.om success
[INFO] create model description success
[INFO] create new context
[INFO] load model ./CosyVoice2-0.5B/flow_static.om success
[INFO] create model description success
[INFO] create new context
[INFO] load model ./CosyVoice2-0.5B/speech_linux_aarch64.om success
[INFO] create model description success
Removing weight norm...
warm up start
0%| | 0/1 [00:00<?, ?it/s][INFO] synthesis text 收到好友从远方寄来的生日礼物，那份意外的惊喜和深深的祝福，让我心中充满了甜蜜的快乐，笑容如花儿般绽放。
....Exception in thread Thread-3 (llm_job):
Traceback (most recent call last):
File "/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_utils/error_code.py", line 43, in wapper
return func(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/dynamo/torchair/core/_backend.py", line 123, in compile
return super(TorchNpuGraph, self).compile()
RuntimeError: EZ9999: Inner Error!
EZ9999: [PID: 489196] 2025-08-07-13:29:21.063.127 numHeads:14 of key must be equal to numHeads:2 of kv when 310P.[FUNC:CheckInputFormatAndLimits][FILE:incre_flash_attention_tiling_check.cc][LINE:320]
TraceBack (most recent call last):
Assert (((funcs->tiling)(reinterpret_cast<gert::TilingContext *>(tiling_context_holder.context_))) == ge::GRAPH_SUCCESS) failed[FUNC:RtParseAndTiling][FILE:op_tiling_rt2.cc][LINE:526]
[GenTask][CalcExtOpRunningParam] CalcTilingSinkRunningParam failed.[FUNC:CalcExtOpRunningParam][FILE:aicore_ops_kernel_builder.cc][LINE:259]
[GenTask][CalcOpRunningParam] CalcExtOpRunningParam failed.[FUNC:CalcOpRunningParam][FILE:aicore_ops_kernel_builder.cc][LINE:227]
Call Calculate op:IncreFlashAttention(IncreFlashAttention) running param failed[FUNC:CalcOpParam][FILE:graph_builder.cc][LINE:211]
[Call][PreRun] Failed, graph_id:1, session_id:1.[FUNC:CompileGraph][FILE:graph_manager.cc][LINE:4545]
[Compile][Graph]Compile graph failed, error code:1343225857, session_id:1, graph_id:1.[FUNC:CompileGraph][FILE:ge_api.cc][LINE:1280]
点赞
收藏
回复
分享
举报
已解决
一、问题现象（附报错日志上下文）：
2025-7-30日的更新17a219f 在300IDUO机器上依然无法使用，报错：
Traceback (most recent call last):
File "/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_utils/error_code.py", line 43, in wapper
return func(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/dynamo/torchair/core/_backend.py", line 123, in compile
return super(TorchNpuGraph, self).compile()
RuntimeError: EZ9999: Inner Error!
EZ9999: [PID: 489196] 2025-08-07-13:29:21.063.127 numHeads:14 of key must be equal to numHeads:2 of kv when 310P.[FUNC:CheckInputFormatAndLimits][FILE:incre_flash_attention_tiling_check.cc][LINE:320]
TraceBack (most recent call last):
Assert (((funcs->tiling)(reinterpret_cast<gert::TilingContext *>(tiling_context_holder.context_))) == ge::GRAPH_SUCCESS) failed[FUNC:RtParseAndTiling][FILE:op_tiling_rt2.cc][LINE:526]
[GenTask][CalcExtOpRunningParam] CalcTilingSinkRunningParam failed.[FUNC:CalcExtOpRunningParam][FILE:aicore_ops_kernel_builder.cc][LINE:259]
[GenTask][CalcOpRunningParam] CalcExtOpRunningParam failed.[FUNC:CalcOpRunningParam][FILE:aicore_ops_kernel_builder.cc][LINE:227]
Call Calculate op:IncreFlashAttention(IncreFlashAttention) running param failed[FUNC:CalcOpParam][FILE:graph_builder.cc][LINE:211]
[Call][PreRun] Failed, graph_id:1, session_id:1.[FUNC:CompileGraph][FILE:graph_manager.cc][LINE:4545]
[Compile][Graph]Compile graph failed, error code:1343225857, session_id:1, graph_id:1.[FUNC:CompileGraph][FILE:ge_api.cc][LINE:1280]
[ERROR] 2025-08-07-13:29:21 (PID:489196, Device:0, RankID:-1) ERR03005 GRAPH internal error
二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x): 8.1.rc1
--Tensorflow/Pytorch/MindSpore 版本: Pytorch: 2.4.0
--Python 版本 (e.g., Python 3.7.5): 3.10.16
--操作系统版本： openEuler release 24.03 (LTS)
三、测试步：
按照 https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/ACL_PyTorch/built-in/audio/CosyVoice/CosyVoice2 的提示进行，其中在运行atc --framework=5 --soc_version=Ascend310P3 --model ./CosyVoice/CosyVoice2-0.5B/speech_token_md.onnx --output ./CosyVoice/CosyVoice2-0.5B/speech --input_shape="feats:1,128,-1;feats_length:1" --precision_mode allow_fp32_to_fp16命令时，有个错误如下，但最后om模型还是转出来了（不知道这个错误对最后有没有影响）：
ATC start working now, please wait for a moment.
Warning: set_tiling_params does not take effect because get_op_mode is dynamic
Warning: set_tiling_params does not take effect because get_op_mode is dynamic
..Warning: set_tiling_params does not take effect because get_op_mode is dynamic
.Warning: set_tiling_params does not take effect because get_op_mode is dynamic
....Warning: set_tiling_params does not take effect because get_op_mode is dynamic
Warning: set_tiling_params does not take effect because get_op_mode is dynamic
....Warning: set_tiling_params does not take effect because get_op_mode is dynamic
....Warning: set_tiling_params does not take effect because get_op_mode is dynamic
..Warning: set_tiling_params does not take effect because get_op_mode is dynamic
..Warning: set_tiling_params does not take effect because get_op_mode is dynamic
...................................................................................................................run stack spill compile error: error: stack frame size (16560) exceeds limit (16000) in function 'te_fused_op_round_sub_add_real_div_mul_add_ad2f18287d99c411b9ad5c422da4ea612176ed93c56480b7c6dfb5cef696cda5_299900002_mix_aiv'
1 error generated.
ATC run success, welcome to the next use.
四、日志信息:
(py3_10_5_21) [root@kylinv241 CosyVoice]# python3 infer.py --model_path=./CosyVoice2-0.5B --stream_out
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:291: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.set_default_device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************
warnings.warn(msg, ImportWarning)
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:246: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/dynamo/torchair/__init__.py:3: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/pkg_resources/__init__.py:3147: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
2025-08-07 13:25:14,582 - modelscope - INFO - PyTorch version 2.4.0 Found.
2025-08-07 13:25:14,584 - modelscope - INFO - Loading ast index from /root/.cache/modelscope/ast_indexer
2025-08-07 13:25:14,640 - modelscope - INFO - Loading done! Current index file version is 1.15.0, with md5 46852a64d9493b37403fd080e09efd06 and a total number of 980 components indexed
failed to import ttsfrd, use WeTextProcessing instead
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/lightning/fabric/__init__.py:41: Deprecated call to `pkg_resources.declare_namespace('lightning.fabric')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/pkg_resources/__init__.py:2559: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('lightning')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(parent)
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/lightning/pytorch/__init__.py:37: Deprecated call to `pkg_resources.declare_namespace('lightning.pytorch')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/pkg_resources/__init__.py:2559: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('lightning')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(parent)
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/diffusers/models/lora.py:393: FutureWarning: `LoRACompatibleLinear` is deprecated and will be removed in version 1.0.0. Use of `LoRACompatibleLinear` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.
deprecate("LoRACompatibleLinear", "1.0.0", deprecation_message)
[INFO] input frame rate=25
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:65: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
warnings.warn(
2025-08-07 13:25:40,980 WETEXT INFO building fst for zh_normalizer ...
[INFO] building fst for zh_normalizer ...
2025-08-07 13:26:22,627 WETEXT INFO done
[INFO] done
2025-08-07 13:26:22,627 WETEXT INFO fst path: /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/zh_tn_tagger.fst
[INFO] fst path: /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/zh_tn_tagger.fst
2025-08-07 13:26:22,627 WETEXT INFO /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/zh_tn_verbalizer.fst
[INFO] /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/zh_tn_verbalizer.fst
2025-08-07 13:26:22,635 WETEXT INFO found existing fst: /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/en_tn_tagger.fst
[INFO] found existing fst: /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/en_tn_tagger.fst
2025-08-07 13:26:22,635 WETEXT INFO /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/en_tn_verbalizer.fst
[INFO] /root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/tn/en_tn_verbalizer.fst
2025-08-07 13:26:22,635 WETEXT INFO skip building fst for en_normalizer ...
[INFO] skip building fst for en_normalizer ...
[WARN] acl repeat initialize
[INFO] acl init success
[INFO] open device 0 success
[INFO] get current context
[INFO] load model ./CosyVoice2-0.5B/flow_linux_aarch64.om success
[INFO] create model description success
[INFO] create new context
[INFO] load model ./CosyVoice2-0.5B/flow_static.om success
[INFO] create model description success
[INFO] create new context
[INFO] load model ./CosyVoice2-0.5B/speech_linux_aarch64.om success
[INFO] create model description success
Removing weight norm...
warm up start
0%| | 0/1 [00:00<?, ?it/s][INFO] synthesis text 收到好友从远方寄来的生日礼物，那份意外的惊喜和深深的祝福，让我心中充满了甜蜜的快乐，笑容如花儿般绽放。
....Exception in thread Thread-3 (llm_job):
Traceback (most recent call last):
File "/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_utils/error_code.py", line 43, in wapper
return func(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_21/lib/python3.10/site-packages/torch_npu/dynamo/torchair/core/_backend.py", line 123, in compile
return super(TorchNpuGraph, self).compile()
RuntimeError: EZ9999: Inner Error!
EZ9999: [PID: 489196] 2025-08-07-13:29:21.063.127 numHeads:14 of key must be equal to numHeads:2 of kv when 310P.[FUNC:CheckInputFormatAndLimits][FILE:incre_flash_attention_tiling_check.cc][LINE:320]
TraceBack (most recent call last):
Assert (((funcs->tiling)(reinterpret_cast<gert::TilingContext *>(tiling_context_holder.context_))) == ge::GRAPH_SUCCESS) failed[FUNC:RtParseAndTiling][FILE:op_tiling_rt2.cc][LINE:526]
[GenTask][CalcExtOpRunningParam] CalcTilingSinkRunningParam failed.[FUNC:CalcExtOpRunningParam][FILE:aicore_ops_kernel_builder.cc][LINE:259]
[GenTask][CalcOpRunningParam] CalcExtOpRunningParam failed.[FUNC:CalcOpRunningParam][FILE:aicore_ops_kernel_builder.cc][LINE:227]
Call Calculate op:IncreFlashAttention(IncreFlashAttention) running param failed[FUNC:CalcOpParam][FILE:graph_builder.cc][LINE:211]
[Call][PreRun] Failed, graph_id:1, session_id:1.[FUNC:CompileGraph][FILE:graph_manager.cc][LINE:4545]
[Compile][Graph]Compile graph failed, error code:1343225857, session_id:1, graph_id:1.[FUNC:CompileGraph][FILE:ge_api.cc][LINE:1280]