香橙派如何进行llama 1b的npu推理加速
收藏回复举报
香橙派如何进行llama 1b的npu推理加速
t('forum.solved') 已解决
发表于2025-01-21 17:08:47
0 查看

直接将transfomers打开的模型导入到npu侧报错如下:

cke_7961.png

Chat with the model! Type 'exit' to end the conversation.

User:

Traceback (most recent call last):

  File "/home/HwHiAiUser/llama/llama-chat.py", line 117, in <module>

    main(args)

  File "/home/HwHiAiUser/llama/llama-chat.py", line 111, in main

    chat_with_model(model, tokenizer, max_new_tokens)

  File "/home/HwHiAiUser/llama/llama-chat.py", line 82, in chat_with_model

    response = generate_text(model, tokenizer, input_to_model, max_new_tokens)

  File "/home/HwHiAiUser/llama/llama-chat.py", line 15, in wrapFunc

    ret = wrap_func(*args, **kwargs)

  File "/home/HwHiAiUser/llama/llama-chat.py", line 47, in generate_text

    output = model.generate(

  File "/usr/local/miniconda3/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context

    return func(*args, **kwargs)

  File "/home/HwHiAiUser/.local/lib/python3.9/site-packages/transformers/generation/utils.py", line 2286, in generate

    result = self._beam_search(

  File "/home/HwHiAiUser/.local/lib/python3.9/site-packages/transformers/generation/utils.py", line 3506, in _beam_search

    outputs = self(**model_inputs, return_dict=True)

  File "/usr/local/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl

    return self._call_impl(*args, **kwargs)

  File "/usr/local/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl

    return forward_call(*args, **kwargs)

  File "/home/HwHiAiUser/.local/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 831, in forward

    outputs = self.model(

  File "/usr/local/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl

    return self._call_impl(*args, **kwargs)

  File "/usr/local/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl

    return forward_call(*args, **kwargs)

  File "/home/HwHiAiUser/.local/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 545, in forward

    inputs_embeds = self.embed_tokens(input_ids)

  File "/usr/local/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl

    return self._call_impl(*args, **kwargs)

  File "/usr/local/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl

    return forward_call(*args, **kwargs)

  File "/usr/local/miniconda3/lib/python3.9/site-packages/torch/nn/modules/sparse.py", line 162, in forward

    return F.embedding(

  File "/usr/local/miniconda3/lib/python3.9/site-packages/torch/nn/functional.py", line 2233, in embedding

    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

RuntimeError: Expected all tensors to be on the same device. Expected NPU tensor, please check whether the input tensor device is correct.

全量代码如下:

我要发帖子