模型训练时报错RuntimeError: aclnnFlashAttentionScoreGetWorkspaceSize call failed, please check!
收藏回复举报
模型训练时报错RuntimeError: aclnnFlashAttentionScoreGetWorkspaceSize call failed, please check!
发表于2025-03-31 17:26:29
0 查看

1 系统环境

硬件环境(Ascend/GPU/CPU): Ascend

MindSpore版本: mindspore=2.4.1

执行模式(PyNative/ Graph):不限

Python版本: Python=3.9

操作系统平台: Ubuntu 22.04

2 报错信息

2.1 问题描述

使用同样的环境和数据,一直正常训练,但是会偶尔出现一次刚开始训练就报错。

2.2 报错信息

RuntimeError: aclnnFlashAttentionScoreGetWorkspaceSize call failed, please check!

---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- EZ1001: 2024-12-04-17:18:01.039.553 Pad InferShape failed.[THREAD:57246] TraceBack (most recent call last): op[PadV3], paddings_contiguous is nullptr![FUNC:PadV3Infershape][FILE:pad_v3.cc][LINE:46][THREAD:57246] infer shape fail, [-1][THREAD:57246] Infer Shape failed.[THREAD:57246] Pad InferShape failed.[THREAD:57246] (Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description) ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/kernel/opapi/aclnn/flash_attention_score_aclnn_kernel.h:51 operator()

3 根因分析

******此处由用户填写******

4 解决方案

******此处由用户填写******

包含文字方案和最终脚本代码

请将正确的脚本打包并上传附件

我要发帖子