Documentation

Sample Call

After plugins of the ONNX framework are developed, you can call Ascend C custom operators on ONNX. The following uses a ONNX framework network that contains only the LeakyReLU operator as an example to describe how to use an inference tool to perform inference, helping you quickly experience the calling process of a custom operator on the network in the inference scenario. (The LeakyReLU operator on the network is mapped to a custom LeakyReLU operator through a plugin.)

Before performing the following steps, you need to develop the custom LeakyReLU operator on the kernel and host, develop the ONNX plugin, and compile and deploy the operator.

For details about the complete example of the LeakyReLU operator implementation, visit this here. For details about the complete example of calling the ONNX framework, visit this here.

  1. Run the following command to obtain the ONNX network model. For reference, the model includes only one LeakyReLU operator.
    wget https://obs-9be7.obs.cn-east-2.myhuaweicloud.com/AscendC/leaky_relu.onnx
  2. Run the following command. (The path and file arguments in the command are for reference only.)
    atc --model=$HOME/module/leaky_relu.onnx --framework=5 --soc_version=<soc_version> --output=$HOME/module/out/leaky_relu --input_shape="X:8,16,1024" --input_format=ND

    The key parameters are described as follows:

    • --model: path of the ONNX network model file (*.onnx).
    • --framework: source framework type. 5 indicates ONNX.
    • --output: path including the file name of the generated offline model. Record this path for future use during application development.
    • --soc_version: Ascend AI Processor version.

      The AI processor model can be obtained in the following ways:

      • Run the npu-smi info command on the server where the Ascend AI Processor is installed to obtain the Chip Name information. The actual value is AscendChip Name. For example, if Chip Name is xxxyy, the actual value is Ascendxxxyy.
    • --input_shape: shape of the input data of the model. Set this parameter based on the shape range supported by the operator and the actual application scenario. In this example, the input X is set to a fixed shape [8, 16, 1024].
    • --input_format: input data format of the model. Set this parameter based on the formats supported by the operator and the actual application scenario. In this example, set this parameter to ND.
  3. Run the export ASCEND_GLOBAL_LOG_LEVEL=1 command to change the log level to INFO. If the following information is displayed, the Ascend C custom operator compilation process is started and the model conversion is successful.
    1
2
3
4
5
    ...
start compile Ascend C operator LeakyReluCustom. kernel name is leaky_relu_custom
compile Ascend C operator: LeakyReluCustom success!
...
ATC run success

    Find the generated offline model (for example, leaky_relu.om) in the path specified by --output.

  4. Prepare the input data in .bin format that meets the model input requirements. Click here to obtain the msame tool, and use it to complete inference by referring to its README document. The following is an example:
    ./msame --model "$HOME/module/out/leaky_relu.om"  --output "$HOME/module/out/" --outfmt TXT --loop 1
Parent topic: ONNX Framework