Introduction to TQue

Tasks communicate and synchronize with each other through queues. TQue is a data structure used to perform queue-related operations and manage related resources. TQue is inherited from the TQueBind parent class. The inheritance relationship is as follows:

Template Parameters

      
           template <TPosition pos, int32_t depth, auto mask = 0> class TQue{...};

Table 1 Parameters in the TQue template

Parameter

Meaning

pos

Logical location of a queue. It can be VECIN, VECOUT, A1, A2, B1, B2, CO1, or CO2. For details about TPosition, see TPosition.

depth

The depth of a queue indicates the number of consecutive enqueue or dequeue operations that can be performed in the queue. During code running, if there are n consecutive enqueue operations (with no dequeue operations in between) in the queue, its depth needs to be set to n.

Note that the queue depth is irrelevant to double buffering. The queue mechanism is used to implement pipeline parallelism. On this basis, double buffering further improves the pipeline utilization. Even if the queue depth is 1, double buffering can still be enabled.

In the non-tensor inplace operation scenario, when the queue depth is set to 1, the compiler is specially optimized for better performance. The recommended value is 1.

In the inplace tensor operation scenario, the value must be set to 0.

In the following example, the queue is not enqueued consecutively, and the queue depth is set to 1.

             
                  a1 = que.AllocTensor(); 
que.EnQue(a1);
a1 = que.DeQue();
que.FreeTensor(a1);

In the following example, the queue is enqueued twice consecutively, and the queue depth should be set to 2. This setting is used only in a few preload scenarios. (For example, two pieces of data are moved in consecutively. After one piece is computed, another piece is moved in, and then the other piece previously moved in is computed.) In other cases, a depth greater than or equal to 2 is not recommended.

             
                  a1 = que.AllocTensor(); 
a2 = que.AllocTensor();
que.EnQue(a1);
que.EnQue(a2);
a1 = que.DeQue();
a2 = que.DeQue(); 
que.FreeTensor(a1);
que.FreeTensor(a2);

mask

If mask is of the int type, bits are used to express information.
- If bit 0 is 1, the data format is converted from ND to NZ. TPosition supports only A1 or B1.
- If bit 1 is 1, the data format is converted from NZ to ND. TPosition supports only CO2.
The following models are supported:

Atlas inference product AI Core

If mask is of the const TQueConfig* type, the structure and parameters of TQueConfig are defined as follows. For a call example, see Example.

             
                  struct TQueConfig {
    bool nd2nz = false;  // true indicates that the data format is converted from ND to NZ. TPosition can only be A1 or B1. The default value is false.
    bool nz2nd = false;  // true indicates that the data format is converted from NZ to ND. TPosition can only be CO2. The default value is false.
    bool scmBlockGroup = false; // TSCM-related parameter. This parameter is reserved. The default value is false.
    uint32_t bufferLen = 0;  // The value must be the same as the value of len entered when InitBuffer is used. Performance can be optimized during compilation. The value 0 indicates that resources are allocated when InitBuffer is used.
    uint32_t bufferNumber = 0;  // The value must be the same as the value of num entered when InitBuffer is used. Performance can be optimized during compilation. The value 0 indicates that resources are allocated when InitBuffer is used.
    uint32_t consumerSize = 0;  // Reserved
    TPosition consumer[8] = {}; // Reserved
    bool enableStaticEvtId = false; // Reserved
    bool enableLoopQueue = false;   // Reserved
};

The following models support the parameters related to ND and NZ format conversion:

Atlas inference product AI Core

TQue Buffer Limit

The buffer allocated by TQue stores the synchronization event ID. Therefore, the number of TQue buffers in the same TPosition is related to the synchronization event ID of hardware.

For the Atlas training product , the number of event IDs is 4.

For the Atlas inference product AI Core, the number of event IDs is 8.

For the Atlas inference product Vector Core, the number of event IDs is 8.

For the Atlas A2 training product / Atlas A2 inference product , the number of event IDs is 8.

For the Atlas A3 training product / Atlas A3 inference product , the number of event IDs is 8.

For the Atlas 200I/500 A2 inference product , the number of event IDs is 8.

The maximum number of buffers of the QUE is 8 or 4, that is, the number of synchronization events that can be inserted is 8 or 4. When the InitBuffer of the TPipe is used to allocate TQue, the maximum number of TQue that can be allocated is 8 or 4 due to buffer limits.

If the number of QUE buffers used at the same time exceeds the limit, no more TQue can be allocated. If you want to continue the allocation, you can call the FreeAllEvent API to release some TQue that is not used temporarily. After the corresponding TQue is used, this API is called to release all events in the corresponding queue. Then, the TQue can be allocated again. Examples are as follows:

Disable double buffering.

        
         
           
           
             // The maximum number of buffers in the VECIN position that can be allocated is 8. If this limit is exceeded, resource allocation may fail when AllocTensor or FreeTensor is used. Therefore, when double buffering is disabled, a maximum of 8 TQue instances can be allocated.
AscendC::TPipe pipe;
int len = 1024;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que0;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que1;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que2;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que3;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que4;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que5;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que6;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que7;
 
pipe.InitBuffer(que0, 1, len);
pipe.InitBuffer(que1, 1, len);
pipe.InitBuffer(que2, 1, len);
pipe.InitBuffer(que3, 1, len);
pipe.InitBuffer(que4, 1, len);
pipe.InitBuffer(que5, 1, len);
pipe.InitBuffer(que6, 1, len);
pipe.InitBuffer(que7, 1, len);

            

          

        
       

Enable double buffering.

        
             // If double buffering is enabled, two buffers are allocated to each TQue. Therefore, a maximum of four TQue instances can be allocated.
AscendC::TPipe pipe;
int len = 1024;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que0;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que1;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que2;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que3;
 
pipe.InitBuffer(que0, 2, len);
pipe.InitBuffer(que1, 2, len);
pipe.InitBuffer(que2, 2, len);
pipe.InitBuffer(que3, 2, len);

Apply for TQue for multiple times.

        
             // If the number of TQue instances reaches the maximum, call FreeAllEvent to allocate more TQue instances.
AscendC::TPipe pipe;
int len = 1024;
AscendC::TQue<AscendC::TPosition::VECIN, 1> que0;
pipe.InitBuffer(que0, 1, len);
AscendC::LocalTensor<half> tensor1 = que0.AllocTensor<half>();
que0.EnQue(tensor1);
tensor1 = que0.DeQue<half>(); // Move the tensor out of the VECOUT queue.
que0.FreeTensor<half>(tensor1);
que0.FreeAllEvent(); // Release all synchronization events of que0. After that, you can continue to allocate TQue.
AscendC::TQue<AscendC::TPosition::VECIN, 1> que1;
pipe.InitBuffer(que1, 1, len);

Example

In the following cases, TQueConfig is transferred to enable the compilation period calculation of bufferNumber. The vector operator does not involve data format conversion. Therefore, the values of ND2NZ and NZ2ND are false.

      
       
         
         
           // Custom meta function for constructing TQueConfig
__aicore__ constexpr AscendC::TQueConfig GetMyTQueConfig(bool nd2nzIn, bool nz2ndIn, bool scmBlockGroupIn,
    uint32_t bufferLenIn, uint32_t bufferNumberIn, uint32_t consumerSizeIn, const AscendC::TPosition consumerIn[])
{
    return {
        .nd2nz = nd2nzIn,
        .nz2nd = nz2ndIn,
        .scmBlockGroup = scmBlockGroupIn,
        .bufferLen = bufferLenIn,
        .bufferNumber = bufferNumberIn,
        .consumerSize = consumerSizeIn,
        .consumer = {consumerIn[0], consumerIn[1], consumerIn[2], consumerIn[3],
            consumerIn[4], consumerIn[5], consumerIn[6], consumerIn[7]}
    };
}
static constexpr AscendC::TPosition tp[8] = {AscendC::TPosition::MAX, AscendC::TPosition::MAX, AscendC::TPosition::MAX, AscendC::TPosition::MAX,
            AscendC::TPosition::MAX, AscendC::TPosition::MAX, AscendC::TPosition::MAX, AscendC::TPosition::MAX};
static constexpr AscendC::TQueConfig conf = GetMyTQueConfig(false, false, false, 0, 1, 0, tp);
...
AscendC::TPipe pipe;
AscendC::TQue<AscendC::TPosition::VECIN, 1, &conf> inQueueX;
AscendC::TQue<AscendC::TPosition::VECOUT, 1, &conf> outQueue;
pipe.InitBuffer(inQueueX, 1, dataSize * sizeof(srcType));
pipe.InitBuffer(outQueue, 1, dataSize * sizeof(int8_t));

          

        

      
     

Parent topic: TQue