Operator Basics
An operator (OP) is a basic unit that performs specific mathematical operations or operations in a deep learning algorithm, such as the activation function (ReLU), convolution (Conv), pooling, and normalization (Softmax). These operators can be combined to build a neural network model.
This section introduces some basic operator terms.
Operator Name
The name of an operator identifies the operator on a network, and therefore must be unique on a network. An example network has operators Conv1, Pool1, and Conv2. Conv1 and Conv2 are of the same convolution type, and each indicates a convolution operation.
Operator Type
Each operator on the network matches the operator implementation file based on operator types, and operators of the same type use the same implementation logic. A network may include the operator composite of the same type. For example, the Conv1 and Conv2 operators in the preceding figure are both convolution operators.
Tensor
A tensor is a container for operators to compute data and contains the following attributes:
Attribute |
Definition |
|---|---|
Shape |
Specifies the shape of a tensor. For example, (10,), (1024, 1024), or (2, 3, 4). For example, the shape (3, 4) indicates a 3 x 4 matrix (3 rows and 4 columns), where the first dimension has three elements, and the second dimension has four elements. Format: (i1, i2, ..., in), where i1 to in are positive integers. |
Data type. |
Specifies the data types of a tensor object. Value range: float16, float32, int8, int16, int32, uint8, uint16, bfloat16, bool, and more |
Format |
Specifies the physical data layout format. For details, see Data Layout Formats. |
Shape
The shape of a tensor is described in the format (D0, D1, ..., Dn – 1), where D0 to Dn are positive integers.
For example, the shape (3, 4) indicates a 3 x 4 matrix, where the first dimension has three elements, and the second dimension has four elements.
The first element of a shape indicates the element count in the outermost square brackets of the tensor, and the second element indicates the element count in the second left square bracket, and so on. The following provides an example.
tensor |
Shape |
Description |
|---|---|---|
1 |
(0,) |
0-dimensional tensor, which is also a scalar |
[1,2,3] |
(3,) |
1-dimensional tensor |
[[1,2],[3,4]] |
(2, 2) |
2-dimensional tensor |
[[[1,2],[3,4]], [[5,6],[7,8]]] |
(2, 2, 2) |
3-dimensional tensor |
How do we understand the physical meaning? Assume that the shape is (4, 20, 20, 3).
There are four pictures, corresponding to 4 in the shape value. Each pixel consists of red, green, and blue colors, that is, the meaning of 3 in the shape value. The width and height of each picture are both 20 pixels. That is, each picture contains 400 pixels (20 x 20).
In programming, the shape can be simply understood as a loop of each layer of a tensor. For example, for operate tensor A with shape (4, 20, 20, 3), the loop statement is as follows.
produce A {
for (i, 0, 4) {
for (j, 0, 20) {
for (p, 0, 20) {
for (q, 0, 3) {
A[((((((i*20) + j)*20) + p)*3) + q)] = a_tensor[((((((i*20) + j)*20) + p)*3) + q)]
}
}
}
}
}
Axis
An axis is denoted by the index of a tensor dimension. For a 2D tensor with five rows and six columns — shape (5, 6) — axis 0 represents the first dimension of the tensor (the rows) and axis 1 represents the second dimension (the columns).
For example, for tensor [[[1,2],[3,4]], [[5,6],[7,8]]] with shape (2, 2, 2), axis 0 represents data in the first dimension, that is, matrices [[1,2],[3,4]] and [[5,6],[7,8]], axis 1 represents data in the second dimension, that is, arrays [1,2], [3,4], [5,6], and [7,8], and axis 2 indicates the data in the third dimension, that is, numbers 1, 2, 3, 4, 5, 6, 7, and 8.
A negative axis is interpreted as indexing from the end.
The axes of an n-dimensional tensor include 0, 1, 2, ..., and n – 1.
