task_time (Task Scheduler Information)
The timeline information of Task Scheduler data is displayed at the Ascend Hardware layer in the msprof_*.json file, and the summary information is summarized in the task_time_*.csv file to identify the scheduling duration during AI task running.
Availability
Atlas 200/500 A2 Inference Product
Atlas Inference Series Product
Atlas Training Series Product
Atlas A2 Training Series Product/Atlas 800I A2 Inference Product
Atlas A3 Training Series Product
Task Scheduler Data in msprof_*.json
The Task Scheduler data in the msprof_*.json file is displayed in each stream of Ascend Hardware. By recording the execution time of each task in different accelerators during AI task running, scheduling durations can be intuitively determined.
The following is an example of the Task Scheduler data in the msprof_*.json file:
See the following table for more details.
Field |
Description |
|---|---|
Title |
API name of a component. |
Start |
Start point on the timeline, which is automatically aligned with that in chrome trace (ms). |
Wall Duration |
Time taken by the calls to an API (ms). |
Task Time(us) |
Task execution duration of an AI CPU operator, in μs. |
Reduce Duration(us) |
Collective communication time of the ALL REDUCE operator, in μs. |
Model Id |
Model ID. |
Task Type |
Type of the accelerator that executes the task, including AI_CORE, AI_VECTOR_CORE, and AI_CPU. |
Stream Id |
ID of the stream where a task is located. The stream ID under Ascend Hardware is the complete logic stream ID of the task, and the stream ID attribute of each API in the timeline on the right is the physical stream ID of the API. |
Task Id |
Task ID. |
Subtask Id |
Subtask ID. |
Aicore Time(ms) |
Theoretical execution time of a task on the AI Core when all blocks are scheduled simultaneously and each block has an equal execution duration, in milliseconds. Typically, the scheduling start time of each block is slightly different, so the value of this field is slightly less than the actual execution time of the task on the AI Core. The data is inaccurate in the manual frequency scaling, Atlas 300V/Atlas 300I Pro, and dynamic frequency scaling (when the power consumption exceeds the default value) scenarios. You are advised not to refer to the data. |
Total Cycle |
Total number of execution cycles of a task on the AI Core, which is the sum of the execution cycles of all blocks. |
Receive Time |
Time when the device receives information about a memory copy task, in μs. This field is displayed only for the MemcopyAsync API. |
Start Time |
Time when a memory copy task starts to copy data, in μs. This field is displayed only for the MemcopyAsync API. |
End Time |
Time when a memory copy task ends to copy data, in us. This field is displayed only for the MemcopyAsync API. |
size(B) |
Amount of data to be copied, in bytes. This field is displayed only for the MemcopyAsync API. |
bandwidth(GB/s) |
Copy bandwidth, in GB/s. This field is displayed only for the MemcopyAsync API. |
operation |
Copy type, such as host-to-device or device-to-host. This field is displayed only for the MemcopyAsync API. |
task_time_*.csv File (Atlas Inference Series Product) (Atlas Training Series Product) (Atlas A2 Training Series Product) (Atlas A3 Training Series Product) (Atlas 200/500 A2 Inference Product)
The file content is formatted as follows.
By examining the operator with highest time consumption in a task, you can determine whether the operator is faulty based on its specific implementation.
Field |
Description |
|---|---|
Device_id |
Device ID. |
kernel_name |
Kernel name. If N/A is displayed, the operator is a non-compute operator. |
kernel_type |
Kernel type, including KERNEL_AICORE and KERNEL_AICPU. |
stream_id |
ID of the stream where a task is located. |
task_id |
Task ID. |
task_time(us) |
Task duration, including scheduling time to the accelerator, execution time on the accelerator, and response end time. The unit is μs. |
task_start(us) |
Task start time, in μs. |
task_stop(us) |
Task end time, in μs. |