ConvConcatFusionPass

Description

Inserts a StridedWrite operator before the concat operator. The inserted StridedWrite, instead of the concat operator, will concatenate Conv2D memory to reduce the performance consumption caused by concat computation.

concat operators include ConcatD and ConcatV2D. Conv2D operators include Conv2D and Conv2D_Compress.

When the subgraph does not contain Dequant or Quant, the following scenarios are involved:

Scenario 1

Scenario 2

Scenario 3:

Scenario 4:

Scenario 5

Scenario 6: When cube and vector operations are not separated on AI Core, Mish fusion is not required.

Scenario 7: When cube and vector operations are separated on AI Core, Mish fusion is required.

When the subgraph does not contain Dequant but contains Quant, the following scenarios are involved:

Scenario 1:

Scenario 2:

Scenario 3:

Scenario 4:

Scenario 5

Scenario 6: When cube and vector operations are not separated on AI Core, the following scenarios are also involved:

When the subgraph contains Dequant but does not contain Quant, the following scenarios are involved:

Scenario 1: When cube and vector operations are not separated on AI Core, Mish fusion is not required.

Scenario 2: When cube and vector operations are separated on AI Core and at least one branch has a Mish operator, Mish fusion is required.

When the subgraph contains both Dequant and Quant, the following scenarios are involved regardless of whether cube and vector operations on AI Core are executed separately or not:

When the subgraph contains both Dequant and Quant and cube and vector operations are not separated on AI Core, the following scenarios are involved:

Restrictions

• In the quantization scenario, the fusion pattern must be enabled. Otherwise, the output dtype of transdata is invalid.
• The dynamic shape scenario is not supported.
• This pass applies when all concat inputs except the final one satisfy channel-dimension alignment.
• Fusion of Quant and Mish operators is allowed when the Concat inputs have their C axis aligned with the data type. That is, when the concat input meets one of the following scenarios, the fusion of the Quant and Mish operators takes effect:
  • If the original dtype is fp16 or float, the value of dim C must be a multiple of 16.
  • If the original dtype is int8, the value of dim C must be a multiple of 32.
  • If the original dtype is int4, the value of dim C must be a multiple of 64.
• If the concat input branch contains the Pooling and mish operators, the quant and mish operators are not fused.
• For details about the conditions for Requant to take effect, see V100RequantFusionPass or V200RequantFusionPass.

Availability

Atlas 200I/500 A2 inference product

Atlas inference product

Atlas training product

Atlas A2 training product/Atlas A2 inference product

Atlas A3 training product/Atlas A3 inference product