启动容器的命令如下: docker run -it --name ai-test -u root -e ASCEND_VISIBLE_DEVICES=1 -v /apprun/ai-custom-nlp:/apprun/ai-custom-nlp efad5dced4f0 /bin/bash
进入容器后训练命令如下(指定了卡1):ASCEND_VISIBLE_DEVICES=1 python train.py
运行还是报错,信息如下:
Traceback (most recent call last):
File "train.py", line 256, in <module>
fine_tuning_nli_cross_encoder(model, data_path, save_path, summary_dir, epoch_num, batch_size)
File "train.py", line 221, in fine_tuning_nli_cross_encoder
summary_writer=summary_writer)
File "/apprun/ai-custom-nlp/src/sentence_transformers/SentenceTransformer.py", line 429, in fit
loss_value = loss_model(features, labels)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/apprun/ai-custom-nlp/src/sentence_transformers/losses/CrossEncoderSoftmaxLoss.py", line 37, in forward
targets_neg = targets_ones - targets_pos
RuntimeError: Expected all tensors to be on the same device. Expected NPU tensor, please check whether the input tensor device is correct.
THPModule_npu_shutdown success.
代码在nvidia卡和寒武纪的卡上都是能跑通的,torch的代码也根据<https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/63RC2alpha002/ptmoddevg/ptmigr/ptmigr_0011.html>已经进行了转换, 唯一的区别是<device = "npu" if torch_npu.npu.is_available() else "cpu"> 这里device只是指定了device = "npu" 没有指定 device = 'npu:{}'.format(args.gpu)
启动容器的命令如下: docker run -it --name ai-test -u root -e ASCEND_VISIBLE_DEVICES=1 -v /apprun/ai-custom-nlp:/apprun/ai-custom-nlp efad5dced4f0 /bin/bash
进入容器后训练命令如下(指定了卡1):ASCEND_VISIBLE_DEVICES=1 python train.py
运行还是报错,信息如下:
Traceback (most recent call last):
File "train.py", line 256, in <module>
fine_tuning_nli_cross_encoder(model, data_path, save_path, summary_dir, epoch_num, batch_size)
File "train.py", line 221, in fine_tuning_nli_cross_encoder
summary_writer=summary_writer)
File "/apprun/ai-custom-nlp/src/sentence_transformers/SentenceTransformer.py", line 429, in fit
loss_value = loss_model(features, labels)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/apprun/ai-custom-nlp/src/sentence_transformers/losses/CrossEncoderSoftmaxLoss.py", line 37, in forward
targets_neg = targets_ones - targets_pos
RuntimeError: Expected all tensors to be on the same device. Expected NPU tensor, please check whether the input tensor device is correct.
THPModule_npu_shutdown success.
代码在nvidia卡和寒武纪的卡上都是能跑通的,torch的代码也根据<https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/63RC2alpha002/ptmoddevg/ptmigr/ptmigr_0011.html>已经进行了转换, 唯一的区别是<device = "npu" if torch_npu.npu.is_available() else "cpu"> 这里device只是指定了device = "npu" 没有指定 device = 'npu:{}'.format(args.gpu)