模型训练报错“Expected NPU tensor, please check whether the input tensor device is correct.”
收藏回复举报
模型训练报错“Expected NPU tensor, please check whether the input tensor device is correct.”
t('forum.solved') 已解决
发表于2023-07-07 18:00:29
0 查看

启动容器的命令如下: docker run -it --name ai-test -u root -e ASCEND_VISIBLE_DEVICES=1 -v /apprun/ai-custom-nlp:/apprun/ai-custom-nlp efad5dced4f0  /bin/bash 

进入容器后训练命令如下(指定了卡1):ASCEND_VISIBLE_DEVICES=1 python train.py

运行还是报错,信息如下:

Traceback (most recent call last): 

  File "train.py", line 256, in <module> 

    fine_tuning_nli_cross_encoder(model, data_path, save_path, summary_dir, epoch_num, batch_size) 

  File "train.py", line 221, in fine_tuning_nli_cross_encoder 

    summary_writer=summary_writer) 

  File "/apprun/ai-custom-nlp/src/sentence_transformers/SentenceTransformer.py", line 429, in fit 

    loss_value = loss_model(features, labels) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl 

    result = self.forward(*input, **kwargs) 

  File "/apprun/ai-custom-nlp/src/sentence_transformers/losses/CrossEncoderSoftmaxLoss.py", line 37, in forward 

    targets_neg = targets_ones - targets_pos 

RuntimeError: Expected all tensors to be on the same device. Expected NPU tensor, please check whether the input tensor device is correct. 

THPModule_npu_shutdown success.

代码在nvidia卡和寒武纪的卡上都是能跑通的,torch的代码也根据<https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/63RC2alpha002/ptmoddevg/ptmigr/ptmigr_0011.html>已经进行了转换, 唯一的区别是<device = "npu" if torch_npu.npu.is_available() else "cpu"> 这里device只是指定了device = "npu" 没有指定 device = 'npu:{}'.format(args.gpu)

本帖最后由 匿名用户2023/07/07 18:00:31 编辑

我要发帖子