DUMP_GRAPH_LEVEL

Description

DUMP_GRAPH_LEVEL sets the graph to dump.

This environment variable can be configured in either of the following ways. Both of them are used to control the number of flushed graphs. You can use them as required. Note that the two configuration methods cannot be used together.

  • Valid values:
    • 1: dumps graphs in all phases.
    • 2: dumps graphs in the trustlist phase. For details about the trustlisted graphs, see the Trustlisted or Not column in Table 1.
    • 3: dumps the last generated graph, that is, the graph optimized and compiled by the GE.
    • 4: dumps the earliest generated graph, that is, the build entry graph provided to the software stack after the GE parses and maps operators. This graph structure has not been compiled or optimized by the GE.
  • Set it to a string separated by vertical bars (|):

    For example, if this environment variable is set to aa|bb, graphs whose names contain aa and bb are dumped. aa and bb must be set to valid character strings during graph build. Valid strings can be obtained from a full dump of the graphs.

DUMP_GRAPH_LEVEL takes effect only when DUMP_GE_GRAPH is enabled. The default value is 2.

Example

  • Set this environment variable to a value:
    export DUMP_GRAPH_LEVEL=1
  • Set this environment variable to a character string separated by vertical bars (|):
    export DUMP_GRAPH_LEVEL="PreRunBegin|AfterInfershape"

Restrictions

  • If this environment variable is set to an invalid value, undefined behavior may occur.
  • If the operator dump data collection function is enabled, see ge.exec.enableDump in Graph Development. Even if the DUMP_GRAPH_LEVEL environment variable is not configured or export DUMP_GRAPH_LEVEL="PreRunBegin|AfterInfershape" is configured but does not contain the Build string, the subgraph ge_proto_xxxx_Build.txt is dumped.
  • This environment variable must be used together with DUMP_GE_GRAPH. That is, when DUMP_GE_GRAPH is enabled, DUMP_GRAPH_LEVEL can be used to control the generated dump graphs. For details about the full dump graph information, see Table 1.
    Table 1 Description of dump graph details

    No.

    Subgraph

    Component

    Phase

    Trustlisted or Not

    1

    ge_proto_xxxx_GraphPreRunBegin.txt

    GE

    Graph before build.

    -

    2

    ge_proto_xxxx_AfterFlowGraphPartition.txt

    GE

    Graph after flow partitioning (flow partitioning is a partitioning mode used in DataFlow).

    -

    3

    ge_proto_xxxx_AfterParallelPartitioner.txt

    GE

    Graph after pipeline parallel partitioning (pipeline here refers to PP in the backend inference scenario).

    -

    4

    ge_proto_xxxx_PreRunBegin.txt

    GE

    Original graph structure.

    Trustlisted

    5

    ge_proto_xxxx_RunCustomPassBeforeInfershape.txt

    GE

    Exit graph of the custom pass before InferShape.

    Trustlisted

    6

    ge_proto_xxxx_PreRunAfterInitPreparation.txt

    FE

    Graph structure after all initialization processing in the graph preparation phase.

    -

    7

    ge_proto_xxxx_PreRunAfterHandleSummaryOp.txt

    GE

    Graph structure after Summary node processing.

    -

    8

    ge_proto_xxxx_PrepareAfterCheckAndUpdateInput.txt

    GE

    Graph structure after the graph input is checked and updated.

    -

    9

    ge_proto_xxxx_PrepareAfterGraphEquivalentTransformation.txt

    GE

    Graph structure after equivalent replacement of For-loop graph structures with While-loop graph structures.

    -

    10

    ge_proto_xxxx_PrepareAfterProcessOutput.txt

    GE

    Graph structure after graph data processing.

    -

    11

    ge_proto_xxxx_PrepareAfterOptimizeAfterGraphNormalization.txt

    GE

    Exit graph after graph optimization following graph normalization.

    -

    12

    ge_proto_xxxx_PrepareAfterInsertAipp.txt

    GE

    Graph structure after AIPP processing (when AIPP parameters are configured).

    -

    13

    ge_proto_xxxx_PrepareAfterProcessAippNodesDataFormat.txt

    GE

    Exit graph after AIPP node format update.

    -

    14

    ge_proto_xxxx_PreRunAfterNormalizeGraph.txt

    GE

    Exit graph after graph normalization.

    Trustlisted

    15

    ge_proto_xxxx_PreRunAfterOptimizeGraphInit.txt

    GE

    Exit graph after graph optimization and initialization.

    -

    16

    ge_proto_xxxx_OptimizeGraph_TagNoConstFoldingAfter.txt

    FE

    Used in quantization scenarios. The FE tags operators that should skip constant folding. During constant folding, the GE checks this tag and skips constant folding if the tag exists.

    -

    17

    ge_proto_xxxx_HcclAfterOptimizeGraphPrepare.txt

    HCCL

    Graph after optimization in the HCCL graph preparation phase.

    -

    18

    ge_proto_xxxx_PreRunAfterOptimizeGraphPrepare.txt

    GE

    Graph structure after original-graph preparation via each operator information library (by calling the OptimizeGraphPrepare API).

    -

    19

    ge_proto_xxxx_PrepareAfterProcessBeforeInfershape.txt

    GE

    Graph structure after dead-edge elimination for conditional operators.

    -

    20

    ge_proto_xxxx_AfterFirstInferformat.txt

    GE

    Graph structure after full-graph format inference.

    -

    21

    ge_proto_xxxx_AfterInfershape.txt

    GE

    Graph structure after full-graph shape inference, accompanied by constant folding.

    Trustlisted

    22

    ge_proto_xxxx_PrepareAfterInferFormatAndShape.txt

    GE

    Graph structure after all InferFormat and InferShape processing. A second full-graph InferFormat occurs between this and the previous graph.

    -

    23

    ge_proto_xxxx_RunCustomPass_AfterInferShape.txt

    GE

    Exit graph of the custom pass after InferShape.

    Trustlisted

    24

    ge_proto_xxxx_AfterSecondInferformat.txt

    GE

    Graph after the second format inference.

    -

    25

    ge_proto_xxxx_PrepareAfterCtrlFlowPreProcess.txt

    GE

    Graph structure after conditional operator preprocessing.

    -

    26

    ge_proto_xxxx_PrepareAfterGetDynamicOutputShape.txt

    GE

    Graph structure after graph output processing in dynamic-profile scenarios.

    -

    27

    ge_proto_xxxx_PrepareAfterProcessAippStage2.txt

    GE

    Graph structure after graph input node processing in AIPP mode.

    -

    28

    ge_proto_xxxx_PrepareAfterPrepareOptimize.txt

    GE

    Graph structure after optimization in the graph preparation phase.

    -

    29

    ge_proto_xxxx_PreRunAfterPrepare.txt

    GE

    Graph structure after all graph preparation processing (same as the previous graph).

    Trustlisted

    30

    ge_proto_xxxx_OptimizeQuantGraph_FeGraphFusionAfter.txt

    FE

    Graph structure after quantization in the graph optimization phase.

    -

    31

    ge_proto_xxxx_OptimizeOriginalGraph_FeGraphFusionAfter.txt

    FE

    Graph structure after graph fusion.

    -

    32

    ge_proto_xxxx_OptimizeOriginalGraph_FeTopoSortingAfter.txt

    FE

    Graph structure after topological sorting following graph fusion, for detecting cycles.

    -

    33

    ge_proto_xxxx_OptimizeOriginalGraph_DSAFeOpJudgeAfter.txt

    FE

    Graph after DataType and Format selection through dynamic shape analysis.

    -

    34

    ge_proto_xxxx_HcclBeforeOptimizeOriginalGraph.txt

    HCCL

    Graph before HCCL-based original-graph optimization.

    -

    35

    ge_proto_xxxx_HcclAfterOptimizeOriginalGraph.txt

    HCCL

    Graph after HCCL-based original-graph optimization.

    -

    36

    ge_proto_xxxx_RunCustomPassAfterBuiltinFusionPass.txt

    GE

    Exit graph of the custom pass after built-in pass execution.

    Trustlisted

    37

    ge_proto_xxxx_PreRunAfterOptimizeOriginalGraph.txt

    GE

    Graph structure after original-graph optimization via each operator information library (by calling the OptimizeOriginalGraph API).

    Trustlisted

    38

    ge_proto_xxxx_PrepareAfterUpdateInputOutputByUserOptions.txt

    GE

    Graph structure after the graph inputs and outputs are processed based on user-specified options.

    -

    39

    ge_proto_xxxx_PrepareAfterUpdateVariableFormats.txt

    GE

    Graph structure after variable Format processing.

    -

    40

    ge_proto_xxxx_PreRunAfterPrepareRunningFormatRefiner.txt

    GE

    Same as the previous graph.

    -

    41

    ge_proto_xxxx_BeforeOptimizeOriginalGraphJudgeInsert.txt

    FE

    Entry graph of the op_judge workflow.

    -

    42

    ge_proto_xxxx_OptimizeOriginalGraph_FeOpDtypeJudgeAfter.txt

    FE

    Graph after precision mode selection.

    -

    43

    ge_proto_xxxx_PreRunAfterRefineRunningPrecision.txt

    GE

    Graph after precision selection.

    -

    44

    ge_proto_xxxx_AfterPrecisionRefine.txt

    GE

    Graph generated after conversion-operator fusion following precision selection.

    -

    45

    ge_proto_xxxx_PreRunAfterAfterPrecisionRefine.txt

    GE

    Graph generated after conversion-operator fusion and automatic fusion following precision selection.

    If automatic fusion is not enabled, this subgraph is identical to

    ge_proto_xxxx_AfterPrecisionRefine.txt.

    -

    46

    ge_proto_xxxx_OptimizeOriginalGraph_FeOpFormatJudgeAfter.txt

    FE

    Full op_judge graph after format selection.

    -

    47

    ge_proto_xxxx_OptimizeOriginalGraph_FeDistHeavyFormatAfter.txt

    FE

    Graph structure after format propagation for heavy operators.

    -

    48

    ge_proto_xxxx_OptimizeOriginalGraph_FeInsertTransNodeAfter.txt

    FE

    Graph structure after conversion operator insertion.

    -

    49

    ge_proto_xxxx_PreRunAfterRefineRunningFormat.txt

    GE

    Graph structure after optimization via each operator information library (by calling the OptimizeOriginalGraphJudgeInsert API).

    -

    50

    ge_proto_xxxx_PreRunAfterSubexpressionMigration.txt

    GE

    Graph structure after common subexpression extraction in dynamic-profile scenarios.

    -

    51

    ge_proto_xxxx_before_SameTransdataBreadthFusionPass.txt

    GE

    Entry graph of SameTransdataBreadthFusionPass.

    -

    52

    ge_proto_xxxx_after_SameTransdataBreadthFusionPass.txt

    GE

    Exit graph of SameTransdataBreadthFusionPass.

    -

    53

    ge_proto_xxxx_OptimizeStage1_1.txt

    GE

    Graph structure after graph optimization stage 1_1.

    -

    54

    ge_proto_xxxx_OptimizeStage1_2.txt

    GE

    Graph structure after graph optimization stage 1_2.

    -

    55

    ge_proto_xxxx_PreRunAfterOptimize1.txt

    GE

    Graph structure after all graph optimization stage 1 processing.

    -

    56

    ge_proto_xxxx_PreRunAfterOptimizeAfterStage1.txt

    GE

    Graph structure after optimization via each operator information library (by calling the OptimizeAfterStage1 API).

    Trustlisted

    57

    ge_proto_xxxx_RunCustomPassAfterOriginGraphOptimize.txt

    GE

    Custom pass executed after original graph optimization.

    Trustlisted

    58

    ge_proto_xxxx_PreRunAfterInferShape2.txt

    GE

    Graph structure after the second InferShape processing.

    -

    59

    ge_proto_xxxx_BeforeStagePartition.txt

    GE

    Graph before stage partitioning.

    -

    60

    ge_proto_xxxx_AfterStagePartition.txt

    GE

    Graph after stage partitioning.

    -

    61

    ge_proto_xxxx_AfterEnginePlacer.txt

    GE

    Graph after engine selection.

    -

    62

    ge_proto_xxxx_Before_DSP.txt

    GE

    Graph before static-dynamic partitioning.

    -

    63

    ge_proto_xxxx_After_DSP.txt

    GE

    Graph after static-dynamic model partitioning.

    -

    64

    ge_proto_xxxx_AfterDynamicShapePartition.txt

    GE

    Graph structure after dynamic-shape graph partitioning.

    -

    65

    ge_proto_xxxx_MergedComputeGraphAfterCompositeEnginePartition.txt

    GE

    Structure of the merged graph after opposite subgraph partitioning and subgraph optimization.

    -

    66

    ge_proto_xxxx_partition0_rank0_inputNodeGraph_AtomicEnginePartitioning.txt

    GE

    Structure of the input node subgraph after graph partitioning based on the atomic engine rules.

    -

    67

    ge_proto_xxxx_partition0_rank1_new_sub_graph1_AtomicEnginePartitioning.txt

    GE

    Structure of subgraph 1 after graph partitioning based on the atomic engine rules.

    -

    68

    ge_proto_xxxx_partition0_rank2_new_sub_graph110_AtomicEnginePartitioning.txt

    GE

    Structure of subgraph 110 after graph partitioning based on the atomic engine rules.

    -

    69

    ge_proto_xxxx_OptimizeSubgraphPreProc.txt

    GE

    Exit graph after subgraph optimization preprocessing.

    -

    70

    ge_proto_xxxx_DNN_VM_RTS_OptimizeSubGraphBefore.txt

    RTS

    -

    -

    71

    ge_proto_xxxx_DNN_VM_RTS_OptimizeSubGraphAfter.txt

    RTS

    -

    -

    72

    ge_proto_xxxx_AIcoreEngine_OptimizeSubGraphBefore.txt

    FE

    Entry graph before AI Core subgraph optimization.

    -

    73

    ge_proto_xxxx_OptimizeSubGraphBefore.txt

    GE

    Subgraph structure before subgraph optimization. Each subgraph has a copy with the same name but different sequence numbers; the total count depends on the number of subgraphs.

    -

    74

    ge_proto_xxxx_OptimizeSubGraphAfter.txt

    GE

    Subgraph structure after subgraph optimization. Each subgraph has a copy with the same name but different sequence numbers; the total count depends on the number of subgraphs.

    -

    75

    ge_proto_xxxx_partition0_rank1_new_sub_graph1_lxfusion_input.txt

    AOE(lxfusion)

    SGAT input graph for ATC and AOE baseline scenarios.

    -

    76

    ge_proto_xxxx_partition0_rank1_new_sub_graph1_after_rebuild.txt

    AOE(lxfusion)

    UB fusion graph within the AOE SGAT internal workflow.

    -

    77

    ge_proto_xxxx_AIcoreEngine_OptimizeSubGraphAfter.txt

    FE

    Exit graph after AI Core subgraph optimization.

    -

    78

    ge_proto_xxxx_OptimizeSubgraphPostProc.txt

    GE

    Exit graph after subgraph optimization postprocessing.

    -

    79

    ge_proto_xxxx_mergedComputeGraph.txt

    GE

    Structure of the merged graph (same as the previous graph).

    -

    80

    ge_proto_xxxx_MergedComputeGraphAfterAtomicEnginePartition.txt

    GE

    Structure of the merged graph after opposite atomic engine partitioning and subgraph optimization.

    -

    81

    ge_proto_xxxx_PreRunAfterOptimizeSubgraph.txt

    GE

    Graph structure after subgraph optimization.

    Trustlisted

    82

    ge_proto_xxxx_OptimizeWholeGraphaicpu_tf_optimizer.txt

    GE

    Graph information after original-graph optimization API calls by each engine (OptimizeWholeGraph followed by the engine name).

    -

    83

    ge_proto_xxxx_OptimizeWholeGraphaicpu_ascend_optimizer.txt

    GE

    Graph information after original-graph optimization API calls by each engine (OptimizeWholeGraph followed by the engine name).

    -

    84

    ge_proto_xxxx_OptimizeWholeGraphdvpp_graph_optimizer.txt

    GE

    Exit graph after DVPP optimization in full-graph optimization.

    -

    85

    ge_proto_xxxx_OptimizeWholeGraphhccl_alltoallvc_fusion_optimizer.txt

    HCCL

    Graph after fusion optimization in the HCCL-based original-graph optimization phase.

    -

    86

    ge_proto_xxxx_OptimizeWholeGraphAIcoreEngine.txt

    GE

    Graph information after original-graph optimization API calls by each engine (OptimizeWholeGraph followed by the engine name).

    -

    87

    ge_proto_xxxx_OptimizeWholeGraphDSAEngine.txt

    FE

    Graph information after original-graph optimization API calls by each engine (OptimizeWholeGraph followed by the engine name).

    -

    88

    ge_proto_xxxx_OptimizeWholeGraphhccl_graph_optimizer.txt

    HCCL

    Graph optimized in the HCCL-based original-graph optimization phase.

    -

    89

    ge_proto_xxxx_OptimizeWholeGraphDNN_VM_RTS_GRAPH_OPTIMIZER_STORE.txt

    GE

    Graph information after original-graph optimization API calls by each engine (OptimizeWholeGraph followed by the engine name).

    -

    90

    ge_proto_xxxx_OptimizeWholeGraphDNN_VM_RTS_FFTS_PLUS_GRAPH_OPTIMIZER_STORE.txt

    RTS

    Graph information after original-graph optimization API calls by each engine (OptimizeWholeGraph followed by the engine name).

    -

    91

    ge_proto_xxxx_OptimizeWholeGraphDNN_VM_HOST_CPU_OPTIMIZER.txt

    GE

    Graph information after original-graph optimization API calls by each engine (OptimizeWholeGraph followed by the engine name).

    -

    92

    ge_proto_xxxx_OptimizeWholeGraphge_local_graph_optimizer.txt

    GE

    Graph information after original-graph optimization API calls by each engine (OptimizeWholeGraph followed by the engine name).

    -

    93

    ge_proto_xxxx_OptimizeWholeGraphffts_plus.txt

    FE

    Graph information after original-graph optimization API calls by each engine (OptimizeWholeGraph followed by the engine name).

    -

    94

    ge_proto_xxxx_PreRunAfterOptimizeWholeGraph.txt

    GE

    Graph structure after optimization via each operator information library (by calling the OptimizeWholeGraph API).

    -

    95

    ge_proto_xxxx_PreRunAfterOptimize2.txt

    GE

    Graph structure after all graph optimization stage 2 processing.

    -

    96

    ge_proto_xxxx_PreRunAfterOptimizeGraphBeforeBuild.txt

    GE

    Entry graph for model build.

    Trustlisted

    97

    ge_proto_xxxx_PreRunAfterOptimizeTensorMove.txt

    GE

    Graph after redundant TensorMove node optimization.

    -

    98

    ge_proto_xxxx_BeforeHandleMemConflict.txt

    GE

    Graph before memory conflict resolution.

    -

    99

    ge_proto_xxxx_AfterHandleMemConflict.txt

    GE

    Graph after memory conflict resolution.

    -

    100

    ge_proto_xxxx_BeforeHandleMemoryLayoutConflict.txt

    GE

    Entry graph before memory layout conflict resolution.

    -

    101

    ge_proto_xxxx_PreRunAfterMemConflictProc.txt

    GE

    Exit graph after memory read/write conflict resolution.

    -

    102

    ge_proto_xxxx_partition0_rank0_inputNodeGraph_SecondPartitioning.txt

    GE

    Structure of the input node subgraph after second partitioning.

    -

    103

    ge_proto_xxxx_partition0_rank1_new_sub_graph1_SecondPartitioning.txt

    GE

    Structure of subgraph 1 after second partitioning.

    -

    104

    ge_proto_xxxx_partition0_rank2_new_sub_graph110_SecondPartitioning.txt

    GE

    Structure of subgraph 110 after second partitioning.

    -

    105

    ge_proto_xxxx_BeforeUnfoldSubgraphs.txt

    GE

    Graph before dynamic-shape graph unfolding.

    -

    106

    ge_proto_xxxx_AfterUnfoldSubgraphs.txt

    GE

    Graph after dynamic-shape graph unfolding.

    -

    107

    ge_proto_xxxx_RunCustomPass_BeforeAssignLogicStream{pass_name}.txt

    GE

    Graph before user-defined stream allocation pass.

    Trustlisted

    108

    ge_proto_xxxx_RunCustomPass_AfterAssignLogicStream{pass_name}.txt

    GE

    Graph after user-defined stream allocation pass.

    Trustlisted

    109

    ge_proto_xxxx_AfterAssignResource.txt

    GE

    Graph after stream allocation, memory allocation, and GenTask.

    -

    110

    ge_proto_xxxx_Build.txt

    GE

    Exit graph for model build.

    Trustlisted

    111

    ge_proto_xxxx_PreRunAfterBuild.txt

    GE

    Same as the previous graph.

    -

    112

    ge_proto_xxxx_BeforeAttrsCompress.txt

    GE

    Graph before offline model compression.

    -

    113

    ge_proto_xxxx_AfterAttrsCompress.txt

    GE

    Graph after offline model compression.

    -

    114

    ge_proto_xxxx_BeforeAttrsDecompress.txt

    GE

    Graph before offline model decompression.

    -

    115

    ge_proto_xxxx_AfterAttrsDecompress.txt

    GE

    Graph after offline model decompression.

    -

    116

    ge_proto_xxxx_ComputeGraphBeforeLowering.txt

    GE

    Computational graph before lowering.

    Trustlisted

    117

    ge_proto_xxxx_Before_MultiStream_LoweringFirstLastEventSync.txt

    GE

    Execution graph before multi-stream event insertion.

    -

    118

    ge_proto_xxxx_ExeGraphBeforeOptimize.txt

    GE

    Execution graph before graph optimization.

    Trustlisted

    119

    ge_proto_xxxx_AfterZeroCopy.txt

    GE

    Execution graph after zero-copy optimization.

    -

    120

    ge_proto_xxxx_AfterCEM.txt

    GE

    Execution graph after CEM optimization.

    -

    121

    ge_proto_xxxx_AfterCopyFlowLaunch.txt

    GE

    Execution graph after inline copy optimization.

    -

    122

    ge_proto_xxxx_TrustOutTensorAfter.txt

    GE

    Execution graph after TrustOutTensor optimization.

    -

    123

    ge_proto_xxxx_AfterAicpuFuseHostInputs.txt

    GE

    Graph after AicpuFuseHostInputs optimization.

    -

    124

    ge_proto_xxxx_ExecuteGraphAfterSplit.txt

    GE

    Final execution graph after dynamic-shape graph splitting.

    Trustlisted

Applicable Models

All processors