attention_score, softmax_max, softmax_sum, softmax_out, seed, offset, numels = torch_npu.npu_fusion_attention(Q_, K_, V_, head_num=8, input_layout="BSH",scale=1 ** 0.5, sync=False) 

  File "/home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/_ops.py", line 692, in __call__ 

    return self._op(*args, **kwargs or {}) 

RuntimeError: Sync:/usr1/02/workspace/j_yxiCvvHE/pytorch/torch_npu/csrc/framework/OpCommand.cpp:162 NPU error, error code is 507015 

[Error]: The aicore execution is abnormal.  

        Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue 

EZ9999: Inner Error! 

EZ9999  The error from device(chipId:2, dieId:0), serial number is 90, there is an fftsplus aivector error exception, core id is 10, error code = 0, dump info: pc start: 0x1240448f73d0, current: 0x1240448ff850, vec error info: 0xff000000ba, mte error info: 0xcc0faff03c, ifu error info: 0x6232488b38980, ccu error info: 0x6260c0b6579c136f, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x124100240080.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1138] 

        TraceBack (most recent call last): 

        The extend info: errcode:(0, 0x800, 0) errorStr: The UB address accessed by the VEC instruction is not aligned. fixp_error0 info: 0xfaff03c, fixp_error1 info: 0xcc fsmId:0, tslot:5, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1150] 

        Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:task_info.cc][LINE:1612] 

        AICORE Kernel task happen error, retCode=0x26.[FUNC:GetError][FILE:stream.cc][LINE:1483] 

        Aicore kernel execute failed, device_id=0, stream_id=29, report_stream_id=29, task_id=45, flip_num=0, fault kernel_name=FlashAttentionScore_32a1d2c7aa995dadcfb93d6011e0c605_high_performance_10000000000000201059_mix_aic, program id=13, hash=8792354494585625866.[FUNC:GetError][FILE:stream.cc][LINE:1483] 

        [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1483] 

        rtStreamSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50] 

        synchronize stream failed, runtime result = 507015[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] 

 

[W NPUStream.cpp:372] Warning: NPU warning, error code is 507015[Error]:  

[Error]: The aicore execution is abnormal.  

        Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue 

EH9999: Inner Error! 

        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50] 

EH9999  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] 

        TraceBack (most recent call last): 

 (function npuSynchronizeDevice) 

[W NPUStream.cpp:372] Warning: NPU warning, error code is 507015[Error]:  

[Error]: The aicore execution is abnormal.  

        Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue 

EH9999: Inner Error! 

        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50] 

EH9999  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] 

        TraceBack (most recent call last): 

 (function npuSynchronizeDevice)