Plugin Development (Caffe)

Overview

This section describes how to develop an operator plugin capable of mapping an operator trained in a third-party framework to one adapted to Ascend AI Processor, and how to register the operator information with GE. During the execution of Caffe-based network, the plugin information in GE is loaded to parse and map the operators in the network to those that can adapt to Ascend AI Processor.

The operators adapted to Ascend AI Processor are referred to as CANN operators.

Currently, one-to-many, many-to-many, and many-to-one mapping from operators in the Caffe framework to CANN operators is not supported.

Principles

The implementation workflow of an operator plugin includes registering the operator types of CANN operators and the operators in the original framework, and mapping the operator attributes in the original framework to attributes of CANN operators. Operator mapping is implemented by the Parser module. Figure 1 shows the plugin implementation workflow in the network execution scenario.

Figure 1 Process for implementing an operator plugin

GE receives a graph, which represents the topology of an original network model trained under a third-party framework. Then, GE is initialized.
GE loads the .so file of the operator plugin from the Register module, which is stored in the opp/built-in/framework/ directory in the CANN component directory.
The Register module reads operator information in the operator plugin .so file and registers the information with the map file of the operator plugin. (Information about all operator plugins is stored in one map file.)
GE requests the Parser module to call the Parser method.
The Parser module obtains the corresponding Parser function from the map file of the operator plugin based on the operator type (OpType), and returns the implementation function ParseParamsByOperatorFn to the Parser module. The Parser module maps the attributes of a third-party operator to attributes of a CANN operator (that is, the attributes defined in the operator prototype). In this way, a third-party operator is mapped to a CANN operator.
Subsequent operations are performed, including graph preparation, partition, and optimization. Then a network model that adapts to Ascend AI Processor is generated.

Plugin Implementation

GE provides the REGISTER_CUSTOM_OP macro to register an operator based on the specified operator name.

The following code shows how to customize a Caffe operator.

#include "register/register.h"
#include "graph/operator.h"
namespace domi
{
REGISTER_CUSTOM_OP("OpType")
    .FrameworkType(CAFFE) 
    .OriginOpType("OriginOpType")
    .ParseParamsByOperatorFn(ParseParamByOpFunc)   // Registers a function for parsing operator attributes.
    .ImplyType(ImplyType::TVM);    // TBE operator: ImplyType::TVM; AI CPU operator: ImplyType::AI_CPU
}

Add the #include command to the beginning of the code implementation file to include the header files related to the plugin implementation functions in the plugin implementation file.
register.h is stored in include/register/ under the CANN software installation directory. Inclusion of this header file enables calls to the operator registration APIs.

operator.h (optional) is stored in include/graph/ under the CANN software installation directory. Inclusion of this header file enables calls to the operator APIs, which can be used to obtain the operator information such as the inputs, outputs, and attributes.
REGISTER_CUSTOM_OP: registers a custom operator. OpType is the type of the operator registered to GE, which must be the same as that in 3.
FrameworkType: specifies the framework type. CAFFE indicates that the original framework is Caffe.
OriginOpType: indicates the type of an operator in the original framework.
ParseParamsByOperatorFn (ParseParamByOpFunc): registers a function for parsing operator attributes. This callback function must be defined by the user.
ParseParamsByOperatorFn is declared as follows:
```
Status ParseParamByOpFunc(const ge::Operator& op_src, ge::Operator& op_dest)
```
- ParseParamByOpFunc: function name, which is user-defined and must be unique.
- op_src: object of class Operator defined by the Caffe framework, including attributes of the custom operator in the Caffe model. The definition is obtained from the Caffe .proto file. If the custom operator is not defined in the caffe.proto file, add the operator definition to the file. For details, see "Operator Project Building" in the readme file. The GetAttr API reads and parses the operator definition from this file.
- op_dest: data structure of the CANN operator, storing operator information. For details about the class Operator, see Operator.
The ParseParamByOpFunc function is implemented as follows:

Call the GetAttr API of class Operator to obtain the attribute value of the op_src object, and then call the SetAttr API to assign the obtained attribute value to the op_dest object.

The GetAttr API supports different types of attribute values. For example:
- For parameters of type int64 in the caffe.proto definition file, call the GetAttr(const char *name, int64_t &attr_value) API to obtain the attribute values.
- For parameters of type enum in the caffe.proto definition file, call the GetAttr(const char *name, int32_t &attr_value) API to obtain the attribute values.
- For parameters of type repeated float in the caffe.proto definition file, call the GetAttr(const char *name, std::vector<float> &attr_value) API to obtain the attribute values.
- For the parameters of the repeated message type defined in the caffe.proto file, use the GetAttr(const char *name, ge::AscendString &attr_value) API to obtain the attribute values. Because attr_value of such type is stored in JSON format during GE processing, the attribute values of the AscendString type need to be cast into that of the String type, and then be converted from strings to the JSON format for parsing.
  An example caffe.proto definition file is as follows.
```
message BiasParameter {
repeated BiasStruct bias_struct = 1;
}
message BiasStruct {
optional uint32  offset = 1;
repeated uint32 width = 2;
};
```
  Use GetAttr("bias_struct", attr_value) to obtain attr_value (AscendString type) and cast it to attr_value (String type), and then convert the String type to the JSON format to access the fields.
  
  Assume that the operator parameters in the .prototxt file are as follows.
```
layer {
name: "bias"
top: "out"
bottom: "data"
type: "Bias"
bias_param {
bias_struct {
offset : 2
width: 8
width: 10
}
bias_struct {
offset : 1
width: 20
}
}
}
```
  The data converted to the JSON format is as follows.
```
{
"bias_struct": [
{
"offset": 2,
"width": [
8,  10
]
}
{
"offset": 1,
"width": [
20
]
}
]
}
```
  You can convert a string into the JSON format in any way that best suits your needs. The following uses json.hpp as an example. You can save the file in any directory under the project path and include the header file for API calls.
  
  The code for reading the JSON fields in the ParseParamByOpFunc function is as follows.
```
if (ge::GRAPH_SUCCESS == op_src.GetAttr("bias_struct", bias_struct_val)) {
  std::string bias_struct_str =  bias_struct_val.GetString();
  // convert to json
  bias_struc_json = nlohmann::json::parse(bias_struct);
  for (int i = 0; i < bias_struc_json["bias_struct"].size(); i++) {
    nlohmann::json bias_value =  bias_struc_json["bias_struct"][i];
    uint32_t value = bias_value["offset"].get<uint32_t>();
    for (int  idx = 0;  idx <  bias_value["width"].size(); idx++) {
      uint32_t value = bias_value["width"][idx].get<uint32_t>();
    }
  }
}
```
  The GetAttr and SetAttr APIs of the current version cannot parse fields of type double or uint64 in the custom.proto file.
  During model conversion using the ATC tool, strong verification is not performed on the obtaining of attributes. When implementing an operator plugin, it is advisable to add the corresponding processing logic for possible GetAttr call failures. For example, return a failure message for a required attribute or prompt the user to set a default value for an optional attribute.
  In the current version, weights of the Convolution and InnerProduct operators cannot be parsed. Therefore, the redefinition of the Convolution and InnerProduct operators is not supported.
ImplyType: specifies the operator implementation type. ImplyType::TVM indicates a TBE operator; ImplyType::AI_CPU indicates an AI CPU operator.

Parent topic: Operator Adaptation