Single-sample inference: If only one sample is input for each inference, this method is called single-sample inference (independent input processing).
Example: Input an image or a sentence to enable the model to predict the result.
Batch inference: If multiple samples are input at a time, the model performs inference on multiple samples at the same time.
Example: Input 32 images or 32 sentences to enable the model to predict all results in parallel.
prof/profiling
Measures the execution details of an operator, including the call stack and call time. The execution details are collected as instructed in Basic msProf Commands in CANN Profiling for performance analysis and tuning.
tiling
In most cases, the local memory cannot completely store the operator input and output. Therefore, you need to transfer some inputs for computation multiple times until the complete final result is obtained. This process of data splitting and block computation is called tiling.
operator
An operator is a logical running unit. The machine learning service encapsulates processing logic such as data loading, data preprocessing, and machine learning algorithms into different logical running units for flexible scheduling. An operator, also called an execution operator, is the minimum unit of a computing task that can be scheduled and executed in the stream computing system.