教程链接： https://www.mindspore.cn/tutorials/zh-CN/r2.2/beginner/train.html

报错信息如下：

Traceback (most recent call last): 

  File "atest.py", line 100, in <module> 

    train_loop(model, train_dataset) 

  File "atest.py", line 76, in train_loop 

    loss = train_step(data, label) 

  File "atest.py", line 68, in train_step 

    (loss, _), grads = grad_fn(data, label) 

  File "/home/ubuntu/.local/lib/python3.7/site-packages/mindspore/ops/composite/base.py", line 625, in after_grad 

    return grad_(fn_, weights)(*args, **kwargs) 

  File "/home/ubuntu/.local/lib/python3.7/site-packages/mindspore/common/api.py", line 121, in wrapper 

    results = fn(*arg, **kwargs) 

  File "/home/ubuntu/.local/lib/python3.7/site-packages/mindspore/ops/composite/base.py", line 600, in after_grad 

    res = self._pynative_forward_run(fn, grad_, weights, args, kwargs) 

  File "/home/ubuntu/.local/lib/python3.7/site-packages/mindspore/ops/composite/base.py", line 650, in _pynative_forward_run 

    outputs = fn(*args, **new_kwargs) 

  File "/home/ubuntu/.local/lib/python3.7/site-packages/mindspore/ops/composite/base.py", line 561, in aux_fn 

    outputs = fn(*args) 

  File "atest.py", line 59, in forward_fn 

    logits = model(data) 

  File "/home/ubuntu/.local/lib/python3.7/site-packages/mindspore/nn/cell.py", line 705, in __call__ 

    raise err 

  File "/home/ubuntu/.local/lib/python3.7/site-packages/mindspore/nn/cell.py", line 702, in __call__ 

    _pynative_executor.end_graph(self, output, *args, **kwargs) 

  File "/home/ubuntu/.local/lib/python3.7/site-packages/mindspore/common/api.py", line 1215, in end_graph 

    self._executor.end_graph(obj, output, *args, *(kwargs.values())) 

RuntimeError: For 'MatMul', encountered an exception:  

---------------------------------------------------- 

- cuBLAS Error: 

---------------------------------------------------- 

cublasGemmEx failed. Possible reasons: the GPU is occupied by other processes. | Error Number: 13 CUBLAS_STATUS_EXECUTION_FAILED: The GPU program failed to execute. This is often caused by a launch failure of the kernel on the GPU, which can be caused by multiple reasons. 

 

---------------------------------------------------- 

- C++ Call Stack: (For framework developers) 

---------------------------------------------------- 

mindspore/ccsrc/plugin/device/gpu/kernel/math/matmul_gpu_kernel.cc:174 LaunchKernel 

mindspore/ccsrc/plugin/device/gpu/kernel/math/matmul_gpu_kernel.cc:162 LaunchKernel 

 when invoke cublas cublasGemmEx 

代码是按照教程复制的

    train_loop(model, train_dataset) 

    test_loop(model, test_dataset, loss_fn)

这两步报错了

不是GPU算力不够的问题 重启过了也是报错