Single-sample inference: If only one sample is input for each inference, this method is called single-sample inference (independent input processing).
Example: Input an image or a sentence and let the model predict the result.
Batch inference: If multiple samples are input at a time, the model performs inference on multiple samples at the same time.
Example: Input 32 images or 32 sentences and let the model predict all results in parallel.
prof/profiling
Measures the execution details of an operator, including the call stack and call time. The execution details are collected as instructed in 《CANN 性能调优工具》中的“msprof采集通用命令” for profiling and tuning.
tiling
Local memory often cannot store all the operator's input and output. Therefore, you need to move parts of the input for computation first, then other parts, until you get the complete final result. This method of splitting data and computing in blocks is called tiling.
operator
An operator is a logical running unit. Machine learning packages tasks such as data loading, preprocessing, and algorithms into different logical units for easy scheduling. An operator, or execution operator, is the smallest schedulable and executable unit in the stream computing system.