Execution Timeout Error of the AI CPU Operator

Symptom

Any of the timeout errors is reported during operator execution.

• Symptom 1
  1. The error code E39999 is reported during Runtime execution. The Runtime error messages "PrintAicpuErrorInfo" and "ErrCode=507018, desc=[aicpu exception]" are printed in the plog file on the host.
  2. In addition, the device log of the AI CPU contains the error message "HandleTaskTimeout".

  This symptom is the same as the error message in Possible Cause > Example 3 in Kernel Execution Error of the AI CPU Operator.

• Symptom 2

  An error is reported during Runtime execution. Runtime error message "PrintAicpuErrorInfo", along with "ErrCode=507017, desc=[aicpu timeout]", is printed in the application log.

  In Ascend EP mode, application run logs are stored in the $HOME/ascend/log/[run|debug]/plog directory by default, and the log file is plog-pid_*.log.

  In Ascend RC mode, application run logs are stored in the $HOME/ascend/log/[run|debug]/device-app-pid directory by default, and the log file is device-app-pid _*.log.

  1	[ERROR] RUNTIME(16243,msame):2022-09-22-11:27:01.794.510 [api_c.cc:661]16243 rtStreamSynchronize:[EXEC][DEFAULT]ErrCode=507017, desc=[aicpu timeout], InnerCode=0x715002a

Possible Cause

• The operator input/output shape is too large, resulting in slow operator execution.
• The hardware performance is insufficient to support the computing of a large number of complex operators.

Solution

Call the aclrtSetOpExecuteTimeOut API to increase the operator execution timeout interval.

The API prototype is defined as follows:

1

aclError aclrtSetOpExecuteTimeOut(uint32_t timeout)      // timeout, in seconds.

If the fault persists, After obtaining the logs, click here to contact technical support.