使用MindSpore的LayerNorm报错ValueError: For 'LayerNorm', gamma or beta shape must match input shape.
收藏回复举报
使用MindSpore的LayerNorm报错ValueError: For 'LayerNorm', gamma or beta shape must match input shape.
发表于2023-09-14 15:58:52
0 查看

1 系统环境

硬件环境(Ascend/GPU/CPU): Ascend/GPU/CPU

MindSpore版本: mindspore=2.0

执行模式(PyNative/ Graph):不限

Python版本: Python=3.7

操作系统平台: 不限

2 报错信息

2.1 问题描述

将begin_norm_axis=1, begin_params_axis=1固定,并未实现与PyTorch完全一致的的功能。

ValueError                                Traceback (most recent call last)
Cell In[12], line 3
      1 add_norm = AddNorm([3, 4], 0.5)
      2 add_norm.set_train(False)
----> 3 add_norm(ops.ones((2, 3, 4)), ops.ones((2, 3, 4))).shape

File ~/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py:664, in Cell.__call__(self, *args, **kwargs)
    662 except Exception as err:
    663     _pynative_executor.clear_res()
--> 664     raise err
    666 if isinstance(output, Parameter):
    667     output = output.data

File ~/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py:661, in Cell.__call__(self, *args, **kwargs)
    659     _pynative_executor.new_graph(self, *args, **kwargs)
    660     output = self._run_construct(args, kwargs)
--> 661     _pynative_executor.end_graph(self, output, *args, **kwargs)
    662 except Exception as err:
    663     _pynative_executor.clear_res()

File ~/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py:1304, in _PyNativeExecutor.end_graph(self, obj, output, *args, **kwargs)
   1291 def end_graph(self, obj, output, *args, **kwargs):
   1292     """
   1293     Clean resources after building forward and backward graph.
   1294 
   (...)
   1302         None.
   1303     """
-> 1304     self._executor.end_graph(obj, output, *args, *(kwargs.values()))

ValueError: For 'LayerNorm', gamma or beta shape must match input shape, but got input shape: [const vector][2, 3, 4], gamma shape: [const vector][3, 4], beta shape: [const vector][3, 4].

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/core/ops/layer_norm.cc:111 InferShape

2.2 脚本代码(代码格式,可上传附件)

class AddNorm(nn.Cell):
    """残差连接后进行层规范化"""
    def __init__(self, normalized_shape, dropout, **kwargs):
        super(AddNorm, self).__init__(**kwargs)
        self.dropout = nn.Dropout(p=dropout)
        self.ln = nn.LayerNorm(normalized_shape) #begin_norm_axis=1, begin_params_axis=1需要增加

    def construct(self, X, Y):
        return self.ln(self.dropout(Y) + X)

add_norm = AddNorm([3, 4], 0.5)
add_norm.set_train(False)
add_norm(ops.ones((2, 3, 4)), ops.ones((2, 3, 4))).shape

完整的代码可以在这里找到:
https://openi.pcl.ac.cn/kewei/d2lkewei-ms/src/branch/master/chapter10_attention-mechanisms/7-transformer-ms.ipynb

3 根因分析

******此处由用户补充详细的定位过程******

4 解决方案

******此处由用户填写******

包含文字方案和最终脚本代码

请将正确的脚本打包并上传附件

我要发帖子