
上面是我的代码,
我已经定位到问题,问题出在了自回归生成text这个地方(将这个及之后的代码注释掉之后,代码可以正常运行),而且,模型可以正常在npu上面执行forward,但是只要一通过
,model.generate来生成就会出错。即使在之前之后加了同步操作也没有用,报错如下
RuntimeError: operator():build/CMakeFiles/torch_npu.dir/compiler_depend.ts:25 NPU function error: c10_npu::acl::AclrtSynchronizeStreamWithTimeout(copy_stream), error code is 507018
[ERROR] 2025-02-27-23:19:18 (PID:287483, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicpu execution is abnormal.
Rectify the fault based on the error information in the ascend log.
E39999: Inner Error!
E39999: [PID: 287483] 2025-02-27-23:19:18.399.829 The error from device(chipId:0, dieId:0), serial number is 11, an exception occurred during AICPU execution, stream_id:1, task_id:1822, errcode:21008, msg:inner error.[FUNC:ProcessStarsAicpuErrorInfo][FILE:device_error_proc.cc][LINE:1479]
TraceBack (most recent call last):
Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
AICPU Kernel task happen error, retCode=0x2a.[FUNC:GetError][FILE:stream.cc][LINE:1103]
Aicpu kernel execute failed, device_id=0, stream_id=1, task_id=1822, errorCode=2a.[FUNC:PrintAicpuErrorInfo][FILE:davinci_kernel_task.cc][LINE:1120]
Aicpu kernel execute failed, device_id=0, stream_id=1, task_id=1822, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1103]
rtStreamSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[W227 23:19:18.498927929 compiler_depend.ts:487] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 287483] 2025-02-27-23:19:18.412.254 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeUsedDevices)
[W227 23:19:18.503732488 compiler_depend.ts:122] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 287483] 2025-02-27-23:19:18.416.990 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function empty_cache)
[W227 23:19:18.508144417 compiler_depend.ts:469] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 287483] 2025-02-27-23:19:18.421.723 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeDevice)
上面是我的代码,
我已经定位到问题,问题出在了自回归生成text这个地方(将这个及之后的代码注释掉之后,代码可以正常运行),而且,模型可以正常在npu上面执行forward,但是只要一通过
,model.generate来生成就会出错。即使在之前之后加了同步操作也没有用,报错如下
RuntimeError: operator():build/CMakeFiles/torch_npu.dir/compiler_depend.ts:25 NPU function error: c10_npu::acl::AclrtSynchronizeStreamWithTimeout(copy_stream), error code is 507018
[ERROR] 2025-02-27-23:19:18 (PID:287483, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicpu execution is abnormal.
Rectify the fault based on the error information in the ascend log.
E39999: Inner Error!
E39999: [PID: 287483] 2025-02-27-23:19:18.399.829 The error from device(chipId:0, dieId:0), serial number is 11, an exception occurred during AICPU execution, stream_id:1, task_id:1822, errcode:21008, msg:inner error.[FUNC:ProcessStarsAicpuErrorInfo][FILE:device_error_proc.cc][LINE:1479]
TraceBack (most recent call last):
Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
AICPU Kernel task happen error, retCode=0x2a.[FUNC:GetError][FILE:stream.cc][LINE:1103]
Aicpu kernel execute failed, device_id=0, stream_id=1, task_id=1822, errorCode=2a.[FUNC:PrintAicpuErrorInfo][FILE:davinci_kernel_task.cc][LINE:1120]
Aicpu kernel execute failed, device_id=0, stream_id=1, task_id=1822, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1103]
rtStreamSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[W227 23:19:18.498927929 compiler_depend.ts:487] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 287483] 2025-02-27-23:19:18.412.254 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeUsedDevices)
[W227 23:19:18.503732488 compiler_depend.ts:122] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 287483] 2025-02-27-23:19:18.416.990 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function empty_cache)
[W227 23:19:18.508144417 compiler_depend.ts:469] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 287483] 2025-02-27-23:19:18.421.723 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeDevice)