MindSpore2.0运行教程所给训练代码样例速度变慢的问题
收藏回复举报
MindSpore2.0运行教程所给训练代码样例速度变慢的问题
新人帖
发表于2022-12-21 18:14:07
0 查看

环境

Linux-x86_64

GPU CUDA 11.4

python 3.7.10

MindSpore 2.0.0 Nightly

问题

在运行教程里的示例代码时,训练速度越来越慢,log如下所示,两个loss之间是一个batch的花费时间。

Downloading data from https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/MNIST_Data.zip (10.3 MB)

file_sizes: 100%|██████████████████████████| 10.8M/10.8M [00:00<00:00, 95.0MB/s]
Extracting zip file...
Successfully downloaded / unzipped to ./
Epoch 1
-------------------------------
loss: 2.303327  [  0/938]
2.0164787769317627
loss: 2.285390  [100/938]
4.2119457721710205
loss: 2.258592  [200/938]
6.720204591751099
loss: 2.186386  [300/938]
10.521805047988892
loss: 1.879479  [400/938]
14.198472023010254
loss: 1.388701  [500/938]
17.38415837287903
loss: 0.997088  [600/938]
21.71060013771057
loss: 0.723973  [700/938]
25.301536321640015
loss: 0.660443  [800/938]
25.50191068649292
loss: 0.592801  [900/938]
29.894129037857056
Test:
 Accuracy: 84.5%, Avg loss: 0.539146

训练输出时间部分代码如下所示:

st = time.time()
for batch, (data, label) in enumerate(dataset.create_tuple_iterator()):
    loss = train_step(data, label)
    if batch % 100 == 0:
        loss, current = loss.asnumpy(), batch
        print(f"loss: {loss:>7f}  [{current:>3d}/{size:>3d}]")
        print(time.time() - st)
        st = time.time()

想请问一下为什么这个训练速度会越来越慢,是我环境或者硬件的问题吗?同时这个模型的运行速度感觉运行也过慢了。

另外我想请问mindspore.value_and_grad是在2.0.0版本新更新的函数吗,我在1.9.0版本的MindSpore下运行示例代码会报错AttributeError: module 'mindspore' has no attribute 'value_and_grad'

我要发帖子