TBE Operator Information Library

Configuration Description

Configure the operator information library file to register the operator implementation information on Ascend AI Processor to the operator information library.

The operator information library file is stored in tbe/op_info_cfg/ai_core/${soc_version}/xx.ini under the custom operator project directory.

Replace ${soc_version} with Ascend AI Processor version in use.

Table 1 describes the configuration rules. Note that the sequences and numbers of inputs, outputs, and attributes must be the same as those in Operator Prototype Definition.

**Table 1** Description of operator information library configurations
Configuration Option	Required	Description
[OpType]	Yes	Operator type, which defines the start of operator information and must be the same as the *OpType* in *REG_OP(OpType)* in Operator Prototype Definition. For details about the OpType naming rules, see Naming Rules for Operator Definition File.
input0.name	Yes	Name of the first input tensor, the same as that in Operator Prototype Definition. If input0.paramType is set to dynamic, set input0.name to x. The name of x must be the same as that in Operator Prototype Definition. At graph run time, x0, x1, x2, and more are automatically generated based on the number of inputs. The inputs are numbered from 0 in ascending order.
input0.paramType	Yes	Type of the first input tensor. dynamic: indicates one or more dynamic inputs, which must be used in conjunction with input0.name. optional: indicates that there could be one input or no input at all. required: indicates that there is only one input. Defaults to required.
input0.dtype	No	Data type (or data types separated with commas) supported by the input tensor. Selected from: float16, float, float32, int8, int16, int32, uint8, uint16, uint32, bool, and more Notes: If the input tensor supports multiple dtype and format combinations, dtype and format must be configured in pairs and in sequence. List all the combinations supported by the operator and separate them with commas (,). Do not insert a space between data types. In the following example, 12 items are configured for input0.dtype and input0.format respectively. input0.dtype=float16,float16,float16,float16,float,float,float,float,int32,int32,int32,int32 input0.format=NCHW,NC1HWC0,NHWC,ND,NCHW,NC1HWC0,NHWC,ND,NCHW,NC1HWC0,NHWC,ND If dynamicFormat.flag is specified to true, this field does not need to be configured as the dtype is inferred by using the Function op_select_format call in the operator implementation file. NOTICE: input0.dtype takes precedence over dynamicFormat if input0.dtype is specified and dynamicFormat.flag is specified to true. You are advised to set either input0.dtype or dynamicFormat based on the dtype inferring mode.
input0.format	No	Format of the first input tensor. If all formats in the original graph are supported, set this parameter to ND. Value range: NCHW, NHWC, NC1HWC0, FRACTAL, ND Notes: If the input tensor supports multiple dtype and format combinations, dtype and format must be configured in pairs and in sequence. List all the combinations supported by the operator and separate them with commas (,). If dynamicFormat.flag is specified to true, input0.format does not need to be configured as the format is inferred by using the Function op_select_format call in the operator implementation file. If op.pattern is configured, FE automatically infers the format for adaptation. As such, input0.format does not need to be configured. NOTICE: The three format-related configuration options are prioritized as follows: input0.format > dynamicFormat > op.pattern That is, the input0.format configuration takes precedence if input0.format and op.pattern are specified and dynamicFormat.flag is specified to true. You are advised to set only one among input0.format, dynamicFormat, and op.pattern based on the format inferring mode.
input0.reshapeType	No	Reshape method for the first input. If the original operator input has dimensions lower than 4, but there is an input format requirement of NC1HWC0, set this parameter to reshape the input to 4D according to the following rules: For 1D to 4D reshape, set reshapeType to the name of the existing dimension (N, C, H, or W). For example, if the original operator input has shape [20] in NHWC format, set reshapeType to N, resulting in shape [20, 1, 1, 1]. For 2D to 4D reshape in NCHW format, set reshapeType to the name of the existing dimensions (NC, CH, HW, CW, NH, or NW) in order. For example, if the original operator input has shape [20, 30], set reshapeType to NC, resulting in shape [20, 30, 1, 1]. For 2D to 4D reshape in NHWC format, set reshapeType to the name of the existing dimensions (NH, HW, WC, NW, NC, or HC) in order. For 3D to 4D reshape, the rules of 2D to 4D reshape also apply. Higher-dimensional reshapeType configuration applies to lower-dimensional inputs. Assume that the original operator input is in NCHW format: If reshapeType is set to NCH, the result shape is [N, 1, 1, 1] when the original input is 1D, [N, C, 1, 1] when the original input is 2D, or [N, C, H, 1] when the original input is 3D. If the original operator input is 4D already, reshapeType does not take effect. If this parameter is not set but there is an input format requirement of NC1HWC0: When the original operator input is in NCHW format: the original 1D input will be reshaped to [1, C, 1, 1]; the original 2D input will be reshaped to, [1, C, H, 1]; the original 3D input will be reshaped to [1, C, H, W]. When the original operator input is in NHWC format: the original 1D input will be reshaped to [1, H, 1, 1]; the original 2D input will be reshaped to [1, H, W, 1]; the original 3D input will be reshaped to [1, H, W, C].
input1.name	No	If the operator has multiple input tensors, add the configurations of input1.xx and input2.xx by referring to the configurations of input0.xx. The inputs are numbered from 0 in ascending order. Name of the input tensor, the same as that in the operator prototype definition.
......	No	The rest configurations for input1. Configure other inputs (if any, for example, input2, input3, ...) of the operator by referring to input0's configurations. NOTICE: If the input dtype and format of the operator are configured, the data types and formats must be the same for all inputs in terms of quantity and sequence and separated by commas (,).
attr.list	No	List of operator attributes required for operator implementation. Separate multiple attributes by commas (,). The attribute sequence must be consistent with the operator declaration. The list is used by FE to obtain the corresponding attributes from OpDesc. Example: stride,padding Note: This parameter needs to be configured if related attributes are required in the operator implementation. Otherwise, the build fails.
attr_key.type	-	The parameter is required if attr.list is set. Otherwise, the build fails. Type of an attribute. attr is a fixed prefix, and key corresponds to a specific parameter. This field must correspond to the attributes in attr.list. The number of attributes in attr.list and the number of attr_key.type records to be configured must be the same. The options are as follows: int: int64 float: float bool: bool str: string listInt: vector<INT> listFloat: vector<FLOAT> listBool: vector<BOOL> listStr: vector<STR> listListInt: vector<vector<int64_t>> type: ge::DataType: all data types listType: vector<ge::DataType> tensor: std::shared_ptr<GeTensor> list tensor: vector<TENSOR>
attr_key.value	-	The parameter is required if attr.list is set. Otherwise, the build fails. Value range of the attribute. Separate multiple values with commas (,). Example: 1,3,5 If all is configured, all attribute values are supported.
attr_key.paramType	No	Whether a parameter is required in OpDesc. For example, in the quantization and non-quantization scenarios, some parameters of the convolution operator may exist or may not exist. The value can be: optional: The parameter is optional. If the value cannot be obtained from OpDesc, no error is reported. required: The parameter is required. If the value cannot be obtained from OpDesc, an error is reported. Defaults to required.
attr_key.defaultValue	No	Default value for attr_key.paramType. Required when attr_key.paramType is set to optional. In this case, if the parameter value fails to be obtained from OpDesc, attr_key.defaultValue is used. Notes: This parameter is invalid when attr_key.paramType is set to required. When attr_key.paramType is optional, if the value fails to be obtained from OpDesc and the defaultValue is not defined, a failure message is returned.
output0.name	No	Name of the first output tensor. If output0.paramType is set to dynamic, set output0.name to y. The name of y must be the same as that in Operator Prototype Definition. At graph run time, y0, y1, y2, and more are automatically generated based on the number of outputs. The outputs are numbered from 0 in ascending order. If the operator has output0, the name parameter of output0 must be configured.
output0.paramType	No	Type of the first output tensor. dynamic: indicates one or more dynamic outputs, which must be used in conjunction with output0.name. optional: indicates that there could be one output or no output at all. required: indicates that there is only one output. Defaults to required.
output0.dtype	No	Data type of the first output. Notes: If the output tensor supports multiple dtype and format combinations, dtype and format must be configured in pairs and in sequence. List all the combinations supported by the operator and separate them with commas (,). In the following example, 12 items are configured for output0.dtype and output0.format respectively. output0.dtype=float16,float16,float16,float16,float,float,float,float,int32,int32,int32,int32 output0.format=NCHW,NC1HWC0,NHWC,ND,NCHW,NC1HWC0,NHWC,ND,NCHW,NC1HWC0,NHWC,ND If dynamicFormat.flag is specified to true, output0.dtype does not need to be configured as the dtype is inferred by using the Function op_select_format call in the operator implementation file. NOTICE: output0.dtype takes precedence over dynamicFormat if output0.dtype is specified and dynamicFormat.flag is specified to true. You are advised to set either output0.dtype or dynamicFormat based on the dtype inferring mode.
output0.format	No	Format of the first output tensor. If all formats in the original graph are supported, set format to ND. Value range: NCHW, NHWC, NC1HWC0, FRACTAL, ND Notes: If the output tensor supports multiple dtype and format combinations, dtype and format must be configured in pairs and in sequence. List all the combinations supported by the operator and separate them with commas (,). If dynamicFormat.flag is specified to true, output0.format does not need to be configured as the format is inferred by using the Function op_select_format call in the operator implementation file. If op.pattern is configured, FE automatically infers the format for adaptation. As such, output0.format does not need to be configured. NOTICE: The three format-related configuration options are prioritized as follows: output0.format > dynamicFormat > op.pattern That is, the output0.format configuration takes precedence if output0.format and op.pattern are specified and dynamicFormat.flag is specified to true. You are advised to set only one among output0.format, dynamicFormat, and op.pattern based on the format inferring mode.
output1.name	No	If the operator has multiple output tensors, add the configurations of output1.xx by referring to the configurations of output0.xx. The outputs are numbered from 0 in ascending order.
opFile.value	No	Name of the operator implementation file. FE locates the operator implementation file based on the file name. If this field is not specified, uppercase letters in the name are replaced with underscores (_) based on the OpType field to match the operator implementation file name. For details about the mapping rule, see Naming Rules for Operator Definition File.
opInterface.value	No	Operator implementation API name. FE calls an operator based on the API name. If this field is not specified, uppercase letters in the name are replaced with underscores (_) based on the OpType field to match the operator API name. For details about the mapping rule, see Naming Rules for Operator Definition File.
op.pattern	No	If op.pattern is specified to any of the following values, the operator's input and output formats do not need to be configured: broadcast: For a broadcast-capable operator, its formats are specified by FE based on the current operator shape. reduce: For a reduce-capable operator, its formats are specified by FE based on the shape, keep_dims, and axis settings of the operator. formatAgnostic: For a format-agnostic operator, for example, the Cast operator that has one input and one output, set op.pattern to formatAgnostic. As such operators support any format, the input and output formats do not need to be enumerated. NOTICE: The op.pattern parameter does not need to be configured when the input and output format lists are configured or dynamicFormat.flag is specified to true, as inputx.format, outputx.format, and dynamicFormat take precedence over op.pattern.
dynamicFormat.flag	No	Defaults to false. If this field is specified to true, the operator's input and output dtype and format are obtained by the Function op_select_format call in the operator implementation file. NOTICE: Beware that the dtype and format lists specified by the following configuration options take precedence over this field. format: corresponds to inputx.format and outputx.format. dtype: corresponds to input0.dtype and output0.dtype.
precision_reduce.flag	No	Operator precision mode during ATC model conversion or network debugging. Valid only when precision_mode is set to allow_mix_precision. If it is set to false, the operator is on the blocklist. In this case, the original data type of the operator must be retained. If it is set to true, the operator is in the trustlist. If the operator supports both float32 and float16 and the original graph format of the operator is float32 or float16, float16 is preferentially selected. If it is not set, the operator is on the graylist. In this case, if this operator has an upstream operator, the data type of the upstream operator is selected. Otherwise, the original data type of the current operator is selected.
heavyOp	No	If it is set to true, heavy format inheritance happens to the operator to reduce the insertion of the Cast operator and improve the network execution efficiency. This parameter is mainly used for Cube operators. According to the inheritance logic, if the input or output format of an operator is one of the following heavy formats: NC1HWC0, C1HWNCoC0, FRACTAL_Z, FRACTAL_NZ, NDC1HWC0, FRACTAL_Z_3D, FRACTAL_Z_3D_TRANSPOSE Recursively search for the connected operator of the input or output, and infer the format of the found operator to the selected heavy format. Stop format inference in the following cases: The connected operator does not support the heavy format of the current operator. The connected operator has the same heavy format as the current operator. The heavyOp field of the connected operator is set to true. Defaults to false.
needCheckSupport.flag	No	Whether to use the check_supported call in the operator implementation file to verify the data type and shape in the operator fusion phase. If this parameter is set to true, FE calls the check_supported API in the operator implementation file to check whether the operator supports the specified input. In this scenario, the Function check_supported needs to be implemented in the operator implementation file. If this parameter is set to false, the verification is not required. Defaults to false.

A configuration example is provided as follows:

[Upsample]
input0.name=x
input0.paramType=required
input0.dtype=float16,float
input0.format=NC1HWC0,NC1HWC0
attr.list=scale,stride_h,stride_w
attr_scale.type=float
attr_scale.value=all
attr_scale.defaultValue=1
attr_scale.paramType=optional
attr_stride_h.type=int
attr_stride_h.value=all
attr_stride_h.defaultValue=2
attr_stride_h.paramType=optional
attr_stride_w.type=int
attr_stride_w.value=all
attr_stride_w.defaultValue=2
attr_stride_w.paramType=optional
output0.name=y
output0.paramType=required
output0.dtype=float16,float
output0.format=NC1HWC0,NC1HWC0

After the custom operator information library file is built and deployed, the operator definition information is stored in <path>/<vendor_name>/op_impl/ai_core/tbe/config/${soc_version}/aic-${soc_version}-ops-info.json of the Ascend AI Processor, where <path> indicates the OPP installation directory (see OPP Deployment) and <vendor_name> indicates the name of the vendor that provides the custom operator. For details about the definition of a vendor name, see Operator Project Building.

Naming Rules for Operator Definition File

Name OpType in upper camel case and separate words with a single capitalized letter.
Name the operator implementation file and operator definition function in either of the following ways:
- To create user-defined names, configure opFile.value and opInterface.value in TBE Operator Information Library.
- If opFile.value and opInterface.value in the TBE Operator Information Library are not configured, FE obtains the operator file name and function name by replacing the OpType as follows.
  The rules are as follows:
  - Replace the first uppercase letter with a lowercase letter.
    Example: Abc -> abc
  - Replace each uppercase letter following lowercase letters with an underscore (_) and the corresponding lowercase letter.
    Example: AbcDef -> abc_def
  - Uppercase letters following a digit or an uppercase letter are regarded as a semantic string. If there is a lowercase letter after this string, replace the last uppercase letter in this string with an underscore (_) and the corresponding lowercase letter, and replace the other uppercase letters with corresponding lowercase letters. If there is no lowercase letter after the string, directly replace the string with lowercase letters.
    Examples: ABCDef -> abc_def; Abc2DEf -> abc2d_ef; Abc2DEF -> abc2def; ABC2dEF -> abc2d_ef

Function op_select_format

If dynamicFormat.flag is set to true, you do not need to configure dtype and format supported by the operator inputs and outputs in the operator information library file. Instead, implement the op_select_format function in the operator implementation file (.py) to infer the dtype and format supported by the operator inputs and outputs.

The op_select_format function is defined as follows:

def op_select_format(input, output, attr, kernel_name="xx"):

The arguments (including the operator inputs, outputs, attributes, and kernel name) to the op_select_format function call must be consistent with those of the operator API call. The op_select_format function returns a string containing the supported input and output formats and data types of the operator.

{
"input0": {
"name": "x",
"dtype": "float16,float16,int8,int8",
"format": "NC1HWC0_C04,NC1HWC0,NC1HWC0_C04,NC1HWC0"
},
"input1": {
"name": "y",
"dtype": "float16,float16,int8,int8",
"format": "FRACTAL_Z_C04,FRACTAL_Z,FRACTAL_Z_C04,FRACTAL_Z"
},
"output0": {
"name": "z",
"dtype": "float16,float16,int32,int32",
"format": "NC1HWC0,NC1HWC0,NC1HWC0,NC1HWC0"
}
}

An implementation sample is provided in Optional Format Inference and Argument Verification.

Function check_supported

If needCheckSupport.flag is set to true, the check_supported API in the operator implementation file is called to verify related information in the operator fusion phase.

The check_supported function is declared as follows:

def check_supported(input_x1, input_x2, output_y, attribute1=None, attribute2=None,..., kernel_name="xx"):

Keep the arguments passed to the check_supported call consistent with those passed to the operator API call (in terms of the operator inputs, outputs, attributes, and kernel name).

Returns True if the verification is passed; otherwise, False.

An implementation sample is provided in Function check_supported.

Parent topic: Operator Information Library Definition