Operator Basics
An operator (OP) is the fundamental unit for executing specific mathematical computations or operations in deep learning algorithms. Common examples include activation functions (such as ReLU), convolution (Conv), pooling, and normalization (such as Softmax). These operators can be combined to build neural network models.
This section introduces some basic operator terms.
Operator Name
An operator's name identifies the operator in a network, and as such it must be unique in the network. An example network has operators Conv1, Pool1, and Conv2. Conv1 and Conv2 are of the same convolution type, and each indicates a convolution operation.
Operator Type
Each operator in the network is implemented according to its operator type, and operators of the same type share the same implementation logic. A network may contain multiple operators of the same type. For example, the Conv1 and Conv2 operators in the preceding figure are both convolution operators.
Tensor
A tensor is a container for operator computation data and has the following attributes.
Attribute |
Definition |
|---|---|
Shape |
Specifies the shape of a tensor. For example, (10,), (1024, 1024), or (2, 3, 4). For example, the shape (3, 4) indicates a 3 x 4 matrix (3 rows and 4 columns), where the first dimension has three elements, and the second dimension has four elements. Format: (i1, i2, …, in), where i1 to in are positive integers. |
Data type |
Specifies the data types of a tensor object. Value range: float16, float32, int8, int16, int32, uint8, uint16, bfloat16, bool, and more |
Data layout format |
Specifies the physical data format. For details, see Data Layout Formats. |
Shape
The shape of a tensor is described in the format (D0, D1, …, Dn – 1), where D0 to Dn are positive integers.
For example, the shape (3, 4) indicates a 3 x 4 matrix (3 rows and 4 columns), where the first dimension has three elements, and the second dimension has four elements.
The first element of a shape indicates the element count in the outermost dimension (the outermost square brackets) of the tensor, and the second element indicates the element count in the next inner dimension (the second set of square brackets from the left), and so on. The following are examples.
Tensor |
Shape |
Description |
|---|---|---|
1 |
(0,) |
0-dimensional tensor, which is also a scalar |
[1,2,3] |
(3,) |
1-dimensional tensor |
[[1,2],[3,4]] |
(2, 2) |
2-dimensional tensor |
[[[1,2],[3,4]], [[5,6],[7,8]]] |
(2, 2, 2) |
3-dimensional tensor |
The tensor shape has a physical meaning. Assume that the shape is (4, 20, 20, 3).
It represents four pictures, corresponding to 4 in the shape. Each pixel consists of red, green, and blue colors, that is, the meaning of 3 in the shape. The width and height of each picture are both 20 pixels. That is, each picture contains 400 pixels (20 x 20).
In programming, the shape can be simply understood as a loop of each layer of a tensor. For example, for operate tensor A with shape (4, 20, 20, 3), the loop statement is as follows.
produce A {
for (i, 0, 4) {
for (j, 0, 20) {
for (p, 0, 20) {
for (q, 0, 3) {
A[((((((i*20) + j)*20) + p)*3) + q)] = a_tensor[((((((i*20) + j)*20) + p)*3) + q)]
}
}
}
}
}
Axis
An axis is denoted by the index of a tensor dimension. For a 2D tensor with five rows and six columns—shape (5, 6)—axis 0 represents the first dimension of the tensor (the rows) and axis 1 represents the second dimension (the columns).
For example, for tensor [[[1,2],[3,4]], [[5,6],[7,8]]] with shape (2, 2, 2), axis 0 represents data in the first dimension, that is, matrices [[1,2],[3,4]] and [[5,6],[7,8]], axis 1 represents data in the second dimension, that is, arrays [1,2], [3,4], [5,6], and [7,8], and axis 2 indicates the data in the third dimension, that is, numbers 1, 2, 3, 4, 5, 6, 7, and 8.
A negative axis is interpreted as indexing from the end.
The axes of an n-dimensional tensor include 0, 1, 2, …, and n – 1.
