Background

Delivery exceptions are a common cause of slow or fast card issues, with typical symptoms as follows:

  • The execution time of a specific operator on a single compute card increases significantly. As shown in the following figure, the time consumed by Dequeue@aclnnLogicalNot is significantly longer than that of the left and right parts (which can be observed through Timeline on MindStudio Insight).
    Figure 1 Unexpected increase in the duration of a single function
  • The overall delivery duration is prolonged. As shown in Figure 2, the delivery duration of card 1 is significantly longer than that of the other card when the same number of operators are executed.
    Figure 2 Slow operator delivery

These kinds of problems are difficult to locate due to complex scenarios and can be classified as delivery exceptions.