model.generate()方法在npu上无法使用
收藏回复举报
model.generate()方法在npu上无法使用
t('forum.solved') 已解决
发表于2025-02-27 23:49:52
0 查看

上面是我的代码,

我已经定位到问题,问题出在了自回归生成text这个地方(将这个及之后的代码注释掉之后,代码可以正常运行),而且,模型可以正常在npu上面执行forward,但是只要一通过

,model.generate来生成就会出错。即使在之前之后加了同步操作也没有用,报错如下

RuntimeError: operator():build/CMakeFiles/torch_npu.dir/compiler_depend.ts:25 NPU function error: c10_npu::acl::AclrtSynchronizeStreamWithTimeout(copy_stream), error code is 507018
[ERROR] 2025-02-27-23:19:18 (PID:287483, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicpu execution is abnormal. 
        Rectify the fault based on the error information in the ascend log.
E39999: Inner Error!
E39999: [PID: 287483] 2025-02-27-23:19:18.399.829 The error from device(chipId:0, dieId:0), serial number is 11, an exception occurred during AICPU execution, stream_id:1, task_id:1822, errcode:21008, msg:inner error.[FUNC:ProcessStarsAicpuErrorInfo][FILE:device_error_proc.cc][LINE:1479]
        TraceBack (most recent call last):
       Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
       AICPU Kernel task happen error, retCode=0x2a.[FUNC:GetError][FILE:stream.cc][LINE:1103]
       Aicpu kernel execute failed, device_id=0, stream_id=1, task_id=1822, errorCode=2a.[FUNC:PrintAicpuErrorInfo][FILE:davinci_kernel_task.cc][LINE:1120]
       Aicpu kernel execute failed, device_id=0, stream_id=1, task_id=1822, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1103]
       rtStreamSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

[W227 23:19:18.498927929 compiler_depend.ts:487] Warning: NPU warning, error code is 507018[Error]: 
[Error]: The aicpu execution is abnormal. 
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 287483] 2025-02-27-23:19:18.412.254 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeUsedDevices)
[W227 23:19:18.503732488 compiler_depend.ts:122] Warning: NPU warning, error code is 507018[Error]: 
[Error]: The aicpu execution is abnormal. 
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 287483] 2025-02-27-23:19:18.416.990 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function empty_cache)
[W227 23:19:18.508144417 compiler_depend.ts:469] Warning: NPU warning, error code is 507018[Error]: 
[Error]: The aicpu execution is abnormal. 
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 287483] 2025-02-27-23:19:18.421.723 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)

我要发帖子