环境： 4.19.90-17.ky10.aarch64 CANN 8.0.0 Python 3.10 torch 2.4.0 torch_npu 2.4.0.post2
(py3_10_5_6) [root@kylinv241 CosyVoice]# python3 infer.py --model_path=${CosyVoice2_0_5B} --stream
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:291: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.set_default_device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************
warnings.warn(msg, ImportWarning)
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:246: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/__init__.py:3: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/pkg_resources/__init__.py:3147: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/pkg_resources/__init__.py:3147: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
2025-05-28 15:19:00,428 - modelscope - INFO - PyTorch version 2.4.0 Found.
2025-05-28 15:19:00,429 - modelscope - INFO - Loading ast index from /root/.cache/modelscope/ast_indexer
2025-05-28 15:19:00,486 - modelscope - INFO - Loading done! Current index file version is 1.15.0, with md5 6d8e11f347d87d9e468fd9aff12969b7 and a total number of 980 components indexed
failed to import ttsfrd, use WeTextProcessing instead
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/lightning/fabric/__init__.py:41: Deprecated call to `pkg_resources.declare_namespace('lightning.fabric')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/pkg_resources/__init__.py:2559: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('lightning')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(parent)
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/lightning/pytorch/__init__.py:37: Deprecated call to `pkg_resources.declare_namespace('lightning.pytorch')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/pkg_resources/__init__.py:2559: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('lightning')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(parent)
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/diffusers/models/lora.py:393: FutureWarning: `LoRACompatibleLinear` is deprecated and will be removed in version 1.0.0. Use of `LoRACompatibleLinear` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.
deprecate("LoRACompatibleLinear", "1.0.0", deprecation_message)
[INFO] input frame rate=25
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:65: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
warnings.warn(
2025-05-28 15:19:23,374 WETEXT INFO building fst for zh_normalizer ...
[INFO] building fst for zh_normalizer ...
2025-05-28 15:20:05,833 WETEXT INFO done
[INFO] done
2025-05-28 15:20:05,834 WETEXT INFO fst path: /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/zh_tn_tagger.fst
[INFO] fst path: /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/zh_tn_tagger.fst
2025-05-28 15:20:05,834 WETEXT INFO /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/zh_tn_verbalizer.fst
[INFO] /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/zh_tn_verbalizer.fst
2025-05-28 15:20:05,848 WETEXT INFO found existing fst: /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/en_tn_tagger.fst
[INFO] found existing fst: /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/en_tn_tagger.fst
2025-05-28 15:20:05,848 WETEXT INFO /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/en_tn_verbalizer.fst
[INFO] /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/en_tn_verbalizer.fst
2025-05-28 15:20:05,848 WETEXT INFO skip building fst for en_normalizer ...
[INFO] skip building fst for en_normalizer ...
[WARN] acl repeat initialize
[INFO] acl init success
[INFO] open device 0 success
[INFO] get current context
[INFO] load model CosyVoice2-0_5B/flow_linux_aarch64.om success
[INFO] create model description success
[INFO] create new context
[INFO] load model CosyVoice2-0_5B/flow_static.om success
[INFO] create model description success
[INFO] create new context
[INFO] load model CosyVoice2-0_5B/speech_linux_aarch64.om success
[INFO] create model description success
Removing weight norm...
warm up start
0%| | 0/1 [00:00<?, ?it/s][INFO] synthesis text 收到好友从远方寄来的生日礼物，那份意外的惊喜和深深的祝福，让我心中充满了甜蜜的快乐，笑容如花儿般绽放。
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_ge_concrete_graph/fx2ge_converter.py:567: UserWarning: When enable frozen_parameter, Parameters will be considered frozen.Please make sure that the Parameters data address remain the same throughout the program runtime.
warnings.warn(f'When enable frozen_parameter, Parameters will be considered frozen.'
0%| | 0/1 [13:53<?, ?it/s]
Traceback (most recent call last):
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_utils/error_code.py", line 43, in wapper
return func(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/core/_backend.py", line 133, in run
return super(TorchNpuGraph, self).run((inputs, assigned_outputs, stream))
RuntimeError: EZ1001: [PID: 3811370] 2025-05-28-15:35:02.876.439 PromptFlashAttention LaunchAicore failed.
TraceBack (most recent call last):
attention mask must be NULL, when Qs,Kvs is unAlign or Qs is not equal to Kvs, Qs = 53, Kvs = 53[FUNC:RunBigKernelTilingWithParams][FILE:prompt_flash_attention_tiling.cpp][LINE:2479]
Tiling failed
Tiling Failed.
Kernel GetWorkspace failed. opType: 18
PromptFlashAttention LaunchAicore failed.
call aclnnPromptFlashAttentionV3 failed:[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:116]
api_ret faild:4294967295[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:119]
Assert ((op_execute_func(op_execute_context)) == ge::SUCCESS) failed[FUNC:ExecuteOpFunc][FILE:aclnn_op_execute_kernel.cc][LINE:47]
Failed to execute rt v2 model for graph graph_1_0, model_id 2.[FUNC:ExecuteWithStreamAsync][FILE:hybrid_model_rt_v2_executor.cc][LINE:1016]
GraphManager ExecuteGrapWithStreamhAsync failed,session id = 1, graph id = 0, stream = 0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:inner_session.cc][LINE:625]
Execute graph with stream async failed, error code:1343225857, session_id:1, graph_id:0, stream:0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:ge_api.cc][LINE:851]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/infer.py", line 51, in <module>
next(cosyvoice.inference_sft(prompt_txt, '中文女', stream=args.stream))
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/cosyvoice/cli/cosyvoice.py", line 73, in inference_sft
for model_output in self.model.tts(**model_input, stream=stream, speed=speed):
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/cosyvoice/cli/model.py", line 368, in tts
for i in self.llm.inference(text=text.to(self.device),
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/cosyvoice/llm/llm.py", line 329, in inference
y_pred, cache = self.llm.forward_one_step(lm_input,
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/cosyvoice/llm/llm.py", line 234, in forward_one_step
outs = self.model(
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/transformers/src/transformers/models/qwen2/modeling_qwen2.py", line 774, in forward
outputs, logits = self.model(
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/transformers/src/transformers/models/qwen2/modeling_qwen2.py", line 456, in forward
return self.cached_prefill(
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/inference/_cache_compiler.py", line 503, in __call__
return self._compiled_model(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/inference/_cache_compiler.py", line 428, in __call__
return self.compiled_func(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 433, in _fn
return fn(*args, **kwargs)
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/transformers/src/transformers/models/qwen2/modeling_qwen2.py", line 520, in prefill
def prefill(
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 600, in _fn
return fn(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 987, in forward
return compiled_fn(full_args)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 217, in runtime_wrapper
all_outs = call_func_at_runtime_with_args(
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 120, in call_func_at_runtime_with_args
out = normalize_as_list(f(args))
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 451, in wrapper
return compiled_fn(runtime_args)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 94, in g
return f(*args)
File "<string>", line 339, in kernel
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/ge/_ge_graph.py", line 659, in run
return self._executor.run(inputs, assigned_outputs, stream)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_utils/error_code.py", line 46, in wapper
raise type(e)("\n".join(msg))
RuntimeError: EZ1001: [PID: 3811370] 2025-05-28-15:35:02.876.439 PromptFlashAttention LaunchAicore failed.
TraceBack (most recent call last):
attention mask must be NULL, when Qs,Kvs is unAlign or Qs is not equal to Kvs, Qs = 53, Kvs = 53[FUNC:RunBigKernelTilingWithParams][FILE:prompt_flash_attention_tiling.cpp][LINE:2479]
Tiling failed
Tiling Failed.
Kernel GetWorkspace failed. opType: 18
PromptFlashAttention LaunchAicore failed.
call aclnnPromptFlashAttentionV3 failed:[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:116]
api_ret faild:4294967295[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:119]
Assert ((op_execute_func(op_execute_context)) == ge::SUCCESS) failed[FUNC:ExecuteOpFunc][FILE:aclnn_op_execute_kernel.cc][LINE:47]
Failed to execute rt v2 model for graph graph_1_0, model_id 2.[FUNC:ExecuteWithStreamAsync][FILE:hybrid_model_rt_v2_executor.cc][LINE:1016]
GraphManager ExecuteGrapWithStreamhAsync failed,session id = 1, graph id = 0, stream = 0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:inner_session.cc][LINE:625]
Execute graph with stream async failed, error code:1343225857, session_id:1, graph_id:0, stream:0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:ge_api.cc][LINE:851]
Traceback (most recent call last):
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_utils/error_code.py", line 43, in wapper
return func(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/core/_backend.py", line 133, in run
return super(TorchNpuGraph, self).run((inputs, assigned_outputs, stream))
RuntimeError: EZ1001: [PID: 3811370] 2025-05-28-15:35:02.876.439 PromptFlashAttention LaunchAicore failed.
TraceBack (most recent call last):
attention mask must be NULL, when Qs,Kvs is unAlign or Qs is not equal to Kvs, Qs = 53, Kvs = 53[FUNC:RunBigKernelTilingWithParams][FILE:prompt_flash_attention_tiling.cpp][LINE:2479]
Tiling failed
Tiling Failed.
Kernel GetWorkspace failed. opType: 18
PromptFlashAttention LaunchAicore failed.
call aclnnPromptFlashAttentionV3 failed:[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:116]
api_ret faild:4294967295[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:119]
Assert ((op_execute_func(op_execute_context)) == ge::SUCCESS) failed[FUNC:ExecuteOpFunc][FILE:aclnn_op_execute_kernel.cc][LINE:47]
Failed to execute rt v2 model for graph graph_1_0, model_id 2.[FUNC:ExecuteWithStreamAsync][FILE:hybrid_model_rt_v2_executor.cc][LINE:1016]
GraphManager ExecuteGrapWithStreamhAsync failed,session id = 1, graph id = 0, stream = 0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:inner_session.cc][LINE:625]
Execute graph with stream async failed, error code:1343225857, session_id:1, graph_id:0, stream:0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:ge_api.cc][LINE:851]
[ERROR] 2025-05-28-15:35:02 (PID:3811370, Device:0, RankID:-1) ERR03005 GRAPH internal error
[INFO] unload model success, model Id is 2147483649
[INFO] unload model success, model Id is 2147483648
[INFO] unload model success, model Id is 1
[WARN] acl repeat destroy
点赞
收藏
回复
分享
举报
已解决
(py3_10_5_6) [root@kylinv241 CosyVoice]# python3 infer.py --model_path=${CosyVoice2_0_5B} --stream
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:291: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.set_default_device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************
warnings.warn(msg, ImportWarning)
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:246: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/__init__.py:3: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/pkg_resources/__init__.py:3147: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/pkg_resources/__init__.py:3147: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
2025-05-28 15:19:00,428 - modelscope - INFO - PyTorch version 2.4.0 Found.
2025-05-28 15:19:00,429 - modelscope - INFO - Loading ast index from /root/.cache/modelscope/ast_indexer
2025-05-28 15:19:00,486 - modelscope - INFO - Loading done! Current index file version is 1.15.0, with md5 6d8e11f347d87d9e468fd9aff12969b7 and a total number of 980 components indexed
failed to import ttsfrd, use WeTextProcessing instead
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/lightning/fabric/__init__.py:41: Deprecated call to `pkg_resources.declare_namespace('lightning.fabric')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/pkg_resources/__init__.py:2559: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('lightning')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(parent)
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/lightning/pytorch/__init__.py:37: Deprecated call to `pkg_resources.declare_namespace('lightning.pytorch')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/pkg_resources/__init__.py:2559: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('lightning')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(parent)
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/diffusers/models/lora.py:393: FutureWarning: `LoRACompatibleLinear` is deprecated and will be removed in version 1.0.0. Use of `LoRACompatibleLinear` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.
deprecate("LoRACompatibleLinear", "1.0.0", deprecation_message)
[INFO] input frame rate=25
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:65: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
warnings.warn(
2025-05-28 15:19:23,374 WETEXT INFO building fst for zh_normalizer ...
[INFO] building fst for zh_normalizer ...
2025-05-28 15:20:05,833 WETEXT INFO done
[INFO] done
2025-05-28 15:20:05,834 WETEXT INFO fst path: /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/zh_tn_tagger.fst
[INFO] fst path: /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/zh_tn_tagger.fst
2025-05-28 15:20:05,834 WETEXT INFO /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/zh_tn_verbalizer.fst
[INFO] /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/zh_tn_verbalizer.fst
2025-05-28 15:20:05,848 WETEXT INFO found existing fst: /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/en_tn_tagger.fst
[INFO] found existing fst: /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/en_tn_tagger.fst
2025-05-28 15:20:05,848 WETEXT INFO /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/en_tn_verbalizer.fst
[INFO] /root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/tn/en_tn_verbalizer.fst
2025-05-28 15:20:05,848 WETEXT INFO skip building fst for en_normalizer ...
[INFO] skip building fst for en_normalizer ...
[WARN] acl repeat initialize
[INFO] acl init success
[INFO] open device 0 success
[INFO] get current context
[INFO] load model CosyVoice2-0_5B/flow_linux_aarch64.om success
[INFO] create model description success
[INFO] create new context
[INFO] load model CosyVoice2-0_5B/flow_static.om success
[INFO] create model description success
[INFO] create new context
[INFO] load model CosyVoice2-0_5B/speech_linux_aarch64.om success
[INFO] create model description success
Removing weight norm...
warm up start
0%| | 0/1 [00:00<?, ?it/s][INFO] synthesis text 收到好友从远方寄来的生日礼物，那份意外的惊喜和深深的祝福，让我心中充满了甜蜜的快乐，笑容如花儿般绽放。
/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_ge_concrete_graph/fx2ge_converter.py:567: UserWarning: When enable frozen_parameter, Parameters will be considered frozen.Please make sure that the Parameters data address remain the same throughout the program runtime.
warnings.warn(f'When enable frozen_parameter, Parameters will be considered frozen.'
0%| | 0/1 [13:53<?, ?it/s]
Traceback (most recent call last):
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_utils/error_code.py", line 43, in wapper
return func(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/core/_backend.py", line 133, in run
return super(TorchNpuGraph, self).run((inputs, assigned_outputs, stream))
RuntimeError: EZ1001: [PID: 3811370] 2025-05-28-15:35:02.876.439 PromptFlashAttention LaunchAicore failed.
TraceBack (most recent call last):
attention mask must be NULL, when Qs,Kvs is unAlign or Qs is not equal to Kvs, Qs = 53, Kvs = 53[FUNC:RunBigKernelTilingWithParams][FILE:prompt_flash_attention_tiling.cpp][LINE:2479]
Tiling failed
Tiling Failed.
Kernel GetWorkspace failed. opType: 18
PromptFlashAttention LaunchAicore failed.
call aclnnPromptFlashAttentionV3 failed:[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:116]
api_ret faild:4294967295[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:119]
Assert ((op_execute_func(op_execute_context)) == ge::SUCCESS) failed[FUNC:ExecuteOpFunc][FILE:aclnn_op_execute_kernel.cc][LINE:47]
Failed to execute rt v2 model for graph graph_1_0, model_id 2.[FUNC:ExecuteWithStreamAsync][FILE:hybrid_model_rt_v2_executor.cc][LINE:1016]
GraphManager ExecuteGrapWithStreamhAsync failed,session id = 1, graph id = 0, stream = 0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:inner_session.cc][LINE:625]
Execute graph with stream async failed, error code:1343225857, session_id:1, graph_id:0, stream:0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:ge_api.cc][LINE:851]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/infer.py", line 51, in <module>
next(cosyvoice.inference_sft(prompt_txt, '中文女', stream=args.stream))
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/cosyvoice/cli/cosyvoice.py", line 73, in inference_sft
for model_output in self.model.tts(**model_input, stream=stream, speed=speed):
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/cosyvoice/cli/model.py", line 368, in tts
for i in self.llm.inference(text=text.to(self.device),
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/cosyvoice/llm/llm.py", line 329, in inference
y_pred, cache = self.llm.forward_one_step(lm_input,
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/cosyvoice/llm/llm.py", line 234, in forward_one_step
outs = self.model(
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/transformers/src/transformers/models/qwen2/modeling_qwen2.py", line 774, in forward
outputs, logits = self.model(
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/transformers/src/transformers/models/qwen2/modeling_qwen2.py", line 456, in forward
return self.cached_prefill(
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/inference/_cache_compiler.py", line 503, in __call__
return self._compiled_model(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/inference/_cache_compiler.py", line 428, in __call__
return self.compiled_func(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 433, in _fn
return fn(*args, **kwargs)
File "/home/ydn/tts_CosyVoice_5_6/CosyVoice_5_9/CosyVoice2/CosyVoice/transformers/src/transformers/models/qwen2/modeling_qwen2.py", line 520, in prefill
def prefill(
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 600, in _fn
return fn(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 987, in forward
return compiled_fn(full_args)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 217, in runtime_wrapper
all_outs = call_func_at_runtime_with_args(
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 120, in call_func_at_runtime_with_args
out = normalize_as_list(f(args))
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 451, in wrapper
return compiled_fn(runtime_args)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 94, in g
return f(*args)
File "<string>", line 339, in kernel
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/ge/_ge_graph.py", line 659, in run
return self._executor.run(inputs, assigned_outputs, stream)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_utils/error_code.py", line 46, in wapper
raise type(e)("\n".join(msg))
RuntimeError: EZ1001: [PID: 3811370] 2025-05-28-15:35:02.876.439 PromptFlashAttention LaunchAicore failed.
TraceBack (most recent call last):
attention mask must be NULL, when Qs,Kvs is unAlign or Qs is not equal to Kvs, Qs = 53, Kvs = 53[FUNC:RunBigKernelTilingWithParams][FILE:prompt_flash_attention_tiling.cpp][LINE:2479]
Tiling failed
Tiling Failed.
Kernel GetWorkspace failed. opType: 18
PromptFlashAttention LaunchAicore failed.
call aclnnPromptFlashAttentionV3 failed:[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:116]
api_ret faild:4294967295[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:119]
Assert ((op_execute_func(op_execute_context)) == ge::SUCCESS) failed[FUNC:ExecuteOpFunc][FILE:aclnn_op_execute_kernel.cc][LINE:47]
Failed to execute rt v2 model for graph graph_1_0, model_id 2.[FUNC:ExecuteWithStreamAsync][FILE:hybrid_model_rt_v2_executor.cc][LINE:1016]
GraphManager ExecuteGrapWithStreamhAsync failed,session id = 1, graph id = 0, stream = 0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:inner_session.cc][LINE:625]
Execute graph with stream async failed, error code:1343225857, session_id:1, graph_id:0, stream:0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:ge_api.cc][LINE:851]
Traceback (most recent call last):
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/_utils/error_code.py", line 43, in wapper
return func(*args, **kwargs)
File "/root/micromamba/envs/py3_10_5_6/lib/python3.10/site-packages/torch_npu/dynamo/torchair/core/_backend.py", line 133, in run
return super(TorchNpuGraph, self).run((inputs, assigned_outputs, stream))
RuntimeError: EZ1001: [PID: 3811370] 2025-05-28-15:35:02.876.439 PromptFlashAttention LaunchAicore failed.
TraceBack (most recent call last):
attention mask must be NULL, when Qs,Kvs is unAlign or Qs is not equal to Kvs, Qs = 53, Kvs = 53[FUNC:RunBigKernelTilingWithParams][FILE:prompt_flash_attention_tiling.cpp][LINE:2479]
Tiling failed
Tiling Failed.
Kernel GetWorkspace failed. opType: 18
PromptFlashAttention LaunchAicore failed.
call aclnnPromptFlashAttentionV3 failed:[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:116]
api_ret faild:4294967295[FUNC:PromptHostExecuteFunc][FILE:fallback_prompt_flash_attention.cpp][LINE:119]
Assert ((op_execute_func(op_execute_context)) == ge::SUCCESS) failed[FUNC:ExecuteOpFunc][FILE:aclnn_op_execute_kernel.cc][LINE:47]
Failed to execute rt v2 model for graph graph_1_0, model_id 2.[FUNC:ExecuteWithStreamAsync][FILE:hybrid_model_rt_v2_executor.cc][LINE:1016]
GraphManager ExecuteGrapWithStreamhAsync failed,session id = 1, graph id = 0, stream = 0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:inner_session.cc][LINE:625]
Execute graph with stream async failed, error code:1343225857, session_id:1, graph_id:0, stream:0xad09c400.[FUNC:ExecuteGraphWithStreamAsync][FILE:ge_api.cc][LINE:851]
[ERROR] 2025-05-28-15:35:02 (PID:3811370, Device:0, RankID:-1) ERR03005 GRAPH internal error
[INFO] unload model success, model Id is 2147483649
[INFO] unload model success, model Id is 2147483648
[INFO] unload model success, model Id is 1
[WARN] acl repeat destroy