Setting a Proper Queue Depth to Make Full Use of Hardware Resources

Background

You can use a media data processing V2 API on the Atlas inference products to set the VPC queue depth within [10, 350]. When user services deliver tasks to a full VPC queue, VPC performs backpressure to block the tasks, which affects performance.

Principles

You can increase the VPC queue depth to address the conflict between user service task delivery and VPC task processing.

• If you deliver more tasks than a VPC channel can handle within a short period of time, the awaited tasks will be cached in the queue without blocking task delivery.
• If you deliver a small number of tasks within a short period of time, VPC will process the tasks cached in the queue to avoid entering the idle state.

You can adjust the queue depth based on the task delivery speed and VPC processing performance of a single channel.

• For a single channel that receives excessive tasks from multiple threads, you are advised to set the queue depth to a larger value.
• For a single channel that receives tasks from one thread and executes the tasks in serial mode, retain the default value since there will be no awaited tasks.

Instructions

After a VPC channel is created using the hi_mpi_vpc_create_chn or hi_mpi_vpc_sys_create_chn call, set the attr parameter in the hi_vpc_chn_attr struct to configure the queue depth.

For details about the sample code for setting the VPC queue depth, see the queue_len parameter in the sample_vpc.cpp file.