task_time (Task Scheduler Information)
The timeline information of Task Scheduler data is displayed at the Ascend Hardware level in the msprof_*.json file, and the summary information is summarized in the task_time_*.csv file to identify the scheduling duration during AI task running.
Applicability
Product |
Supported |
|---|---|
Atlas 350 Accelerator Card |
√ |
√ |
|
√ |
|
√ |
|
√ |
|
√ |
|
Atlas 350 Accelerator Card |
√ |
Task Scheduler Data in msprof_*.json
The Task Scheduler data in the msprof_*.json file is displayed in each stream of Ascend Hardware. By recording the execution time of each task in different accelerators during AI task running, scheduling durations can be intuitively determined.
The following is an example of the Task Scheduler data in the msprof_*.json file:
See the following table for more details.
Field |
Description |
|---|---|
Title |
API name of a component. |
Start |
Start point on the timeline, which is automatically aligned with that in chrome trace (ms). |
Wall Duration |
Time taken by the calls to an API (ms). |
Task Time(us) |
Task execution duration of an AI CPU operator, in μs. |
Reduce Duration(us) |
Collective communication time of the ALL REDUCE operator, in μs. |
Model Id |
Model ID. |
Task Type |
Type of the accelerator that executes the task, such as AI_CORE, AI_VECTOR_CORE, or AI_CPU. |
Stream Id |
ID of the stream where a task is located. The stream ID under Ascend Hardware is the complete logic stream ID of the task, and the stream ID attribute of each API in the timeline on the right is the physical stream ID of the API. |
Task Id |
Task ID. |
Subtask Id |
Subtask ID. |
Aicore Time(ms) |
Theoretical execution time of a task on the AI Core when all blocks are scheduled simultaneously and each block has an equal execution duration. The unit is ms. Typically, the scheduling start time of each block is slightly different, so the value of this field is slightly less than the actual execution time of the task on the AI Core. The data is inaccurate in the manual frequency modulation, dynamic frequency modulation (the power consumption exceeds the default value), and Atlas 300V/Atlas 300I Pro scenarios. You are not advised referring to it. |
Total Cycle |
Total number of execution cycles of a task on the AI Core, which is the sum of the execution cycles of all blocks. |
Receive Time |
Time when the device receives information about a memory copy task, in μs. This field is displayed only for the MemcopyAsync API. |
Start Time |
Time when a memory copy task starts to copy data, in μs. This field is displayed only for the MemcopyAsync API. |
End Time |
Time when a memory copy task ends to copy data, in us. This field is displayed only for the MemcopyAsync API. |
size(B) |
Amount of data to be copied, in bytes. This field is displayed only for the MemcopyAsync API. |
bandwidth(GB/s) |
Copy bandwidth, in GB/s. This field is displayed only for the MemcopyAsync API. |
operation |
Copy type, such as host-to-device or device-to-host. This field is displayed only for the MemcopyAsync API. |
task_time_*.csv File (Atlas inference product ) (Atlas training product ) (Atlas A2 training products ) (Atlas A3 training product /Atlas A3 inference product ) (Atlas 200I/500 A2 inference product ) (Atlas 350 Accelerator Card)
The following shows the file content format.
By examining the operator with the highest time consumption in a task, you can determine whether the operator is faulty based on its specific implementation.
Field |
Description |
|---|---|
Device_id |
Device ID. |
kernel_name |
Kernel name. If N/A is displayed, the operator is a non-compute operator. |
kernel_type |
Kernel type, including KERNEL_AICORE, KERNEL_AICPU, and more. |
stream_id |
ID of the stream where a task is located. |
task_id |
Task ID. |
task_time(us) |
Task duration, including scheduling time to the accelerator, execution time on the accelerator, and response end time. The unit is μs. |
task_start(us) |
Task start time, in μs. |
task_stop(us) |
Task end time, in μs. |