Sample Call

After plugins of the ONNX framework are developed, you can call Ascend C custom operators on ONNX. The following uses a ONNX framework network that contains only the LeakyReLU operator as an example to describe how to use an inference tool to perform inference, helping you quickly experience the calling process of a custom operator on the network in the inference scenario. (The LeakyReLU operator on the network is mapped to a custom LeakyReLU operator through a plugin.)

Before performing the following steps, you need to develop the custom LeakyReLU operator on the kernel and host, develop the ONNX plugin, and compile and deploy the operator.

For details about the complete example of the LeakyReLU operator implementation, visit this LINK. For details about the complete example of calling the ONNX framework, visit this LINK.

1. Run the following command to obtain the ONNX network model. For reference, the model includes only one LeakyReLU operator.

   wget https://obs-9be7.obs.cn-east-2.myhuaweicloud.com/AscendC/leaky_relu.onnx

2. Run the following command. (The path and file arguments in the command are for reference only.)

   atc --model=$HOME/module/leaky_relu.onnx --framework=5 --soc_version=<soc_version> --output=$HOME/module/out/leaky_relu --input_shape="X:8,16,1024" --input_format=ND

   The key parameters are described as follows:

   • --model: path of the ONNX network model file (*.onnx).
   • --framework: source framework type. 5 indicates ONNX.
   • --output: path and file name of the generated offline model. Record this path for future use during application development.
   • --soc_version: AI processor version.
     

     The AI processor model can be obtained in the following ways:

     • For the following products: Run the npu-smi info command on the server where AI processor is installed to obtain the Name information. The actual value is AscendName. For example, if Name is xxxyy, the actual value is Ascendxxxyy.

       Atlas A2 training product/Atlas A2 inference product

       Atlas 200I/500 A2 inference product

       Atlas inference product

       Atlas training product

     • For the Atlas A3 training product/Atlas A3 inference product: Run the npu-smi info -t board -i id -c chip_id command on the server where AI processor is installed to obtain the Chip Name and NPU Name information. The actual value is Chip Name_NPU Name. For example, if the value of Chip Name is Ascendxxx and the value of NPU Name is 1234, the actual value is Ascendxxx_1234. Note that:
       • id: device ID, which is the NPU ID obtained by running the npu-smi info -l command.
       • chip_id: chip ID, which is obtained by running the npu-smi info -m command.
     • For the Atlas 350 Accelerator Card: Run the npu-smi info -t board -i id command on the server where AI processor is installed to obtain the Chip Name and NPU Name information. The actual value is Chip Name_NPU Name. For example, if the value of Chip Name is Ascendxxx and the value of NPU Name is 1234, the actual value is Ascendxxx_1234.

       id indicates the device ID, which is the NPU ID obtained by running the npu-smi info -l command.

   • --input_shape: shape of the input data of the model. Set this parameter based on the shape range supported by the operator and the actual application scenario. In this example, the input X is set to a fixed shape [8, 16, 1024].
   • --input_format: input data format of the model. Set this parameter based on the formats supported by the operator and the actual application scenario. In this example, set this parameter to ND.
3. Run the export ASCEND_GLOBAL_LOG_LEVEL=1 command to change the log level to INFO. If the following information is displayed, the Ascend C custom operator build process is started and the model conversion is successful.

   1 2 3 4 5

   ... start compile Ascend C operator LeakyReluCustom. kernel name is leaky_relu_custom compile Ascend C operator: LeakyReluCustom success! ... ATC run success

   Find the generated offline model (for example, leaky_relu.om) in the path specified by --output.

4. Load the model in the application by using APIs such as aclmdlExecute to perform inference.