DeepseekV3的预训练流程，缩减层数和并行策略，但是发现走不通，报flash attn算子调用失败问题，不确定是不是因为缩减的问题...

 

[rank3]:[E compiler_depend.ts:416] call aclnnFlashAttentionVarLenScore failed, detail:E89999: Inner Error! 

E89999: [PID: 1525409] 2025-04-18-11:04:30.223.802 key or value shape is invalid[FUNC:CheckParams][FILE:flash_attention_score_tiling.cpp][LINE:87] 

        TraceBack (most recent call last): 

       Tiling failed 

       Tiling Failed. 

       Kernel GetWorkspace failed. opType: 37 

       Kernel Run failed. opType: 37, FlashAttentionScore 

       launch failed for FlashAttentionScore, errno:561103. 

 

[ERROR] 2025-04-18-11:04:30 (PID:1525409, Device:3, RankID:3) ERR01100 OPS call acl api failed 

Exception raised from operator() at build/CMakeFiles/torch_npu.dir/compiler_depend.ts:1561 (most recent call first): 

frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xfffdebca2ee8 in /root/miniconda3/envs/work/lib/python3.9/site-packages/torch/lib/libc10.so) 

frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xfffdebc4fd10 in /root/miniconda3/envs/work/lib/python3.9/site-packages/torch/lib/libc10.so) 

frame #2: <unknown function> + 0xc84004 (0xfffddf5f4004 in /root/miniconda3/envs/work/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so) 

frame #3: <unknown function> + 0x1537254 (0xfffddfea7254 in /root/miniconda3/envs/work/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so) 

frame #4: <unknown function> + 0x7bec84 (0xfffddf12ec84 in /root/miniconda3/envs/work/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so) 

frame #5: <unknown function> + 0x7bf3f0 (0xfffddf12f3f0 in /root/miniconda3/envs/work/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so) 

frame #6: <unknown function> + 0x7bc64c (0xfffddf12c64c in /root/miniconda3/envs/work/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so) 

frame #7: <unknown function> + 0xb967c (0xfffdebcd967c in /root/miniconda3/envs/work/lib/python3.9/site-packages/torch/lib/libc10.so) 

frame #8: <unknown function> + 0x7d5c8 (0xfffdf5add5c8 in /lib/aarch64-linux-gnu/libc.so.6) 

frame #9: <unknown function> + 0xe5edc (0xfffdf5b45edc in /lib/aarch64-linux-gnu/libc.so.6) 

[rank3]: Traceback (most recent call last): 

[rank3]:   File "/work/share/zjc/ms-llm/MindSpeed-LLM/pretrain_gpt.py", line 245, in <module> 

[rank3]:     main() 

[rank3]:   File "/work/share/zjc/ms-llm/MindSpeed-LLM/pretrain_gpt.py", line 238, in main 

[rank3]:     pretrain(train_valid_test_datasets_provider, 

[rank3]:   File "/work/share/zjc/ms-llm/MindSpeed-LLM/mindspeed_llm/training/training.py", line 404, in pretrain 

[rank3]:     iteration, num_floating_point_operations_so_far = train(*train_args) 

[rank3]:   File "/work/share/zjc/ms-llm/MindSpeed-LLM/mindspeed_llm/training/training.py", line 553, in train 

[rank3]:     train_step(forward_step_func, 

[rank3]:   File "/root/miniconda3/envs/work/lib/python3.9/site-packages/mindspeed/core/pipeline_parallel/dualpipev/dualpipev_chunks.py", line 130, in train_step 

[rank3]:     losses_reduced = forward_backward_func( 

[rank3]:   File "/root/miniconda3/envs/work/lib/python3.9/site-packages/mindspeed/core/pipeline_parallel/dualpipev/dualpipev_schedules.py", line 855, in forward_backward_pipelining_with_cutinhalf 

[rank3]:     output_tensor_warmup, _ = forward_step_helper(master_chunk_id, master_cur_microbatch, checkpoint_activations_microbatch, 

[rank3]:   File "/root/miniconda3/envs/work/lib/python3.9/site-packages/mindspeed/core/pipeline_parallel/dualpipev/dualpipev_schedules.py", line 790, in forward_step_helper 

[rank3]:     output_tensor, num_tokens = forward_step_with_model_graph( 

[rank3]:   File "/root/miniconda3/envs/work/lib/python3.9/site-packages/mindspeed/core/pipeline_parallel/dualpipev/dualpipev_schedules.py", line 611, in forward_step_with_model_graph 

[rank3]:     output_tensor, loss_func = pretrain_gpt_forward_step_dualpipe( 

[rank3]:   File "/root/miniconda3/envs/work/lib/python3.9/site-packages/mindspeed/core/pipeline_parallel/dualpipev/dualpipev_schedules.py", line 339, in pretrain_gpt_forward_step_dualpipe 

[rank3]:     tokens, labels, loss_mask, attention_mask, position_ids = get_batch( 

[rank3]:   File "/work/share/zjc/ms-llm/MindSpeed-LLM/pretrain_gpt.py", line 116, in get_batch 

[rank3]:     generate_actual_seq_len(batch) 

[rank3]:   File "/work/share/zjc/ms-llm/MindSpeed-LLM/mindspeed_llm/training/utils.py", line 62, in generate_actual_seq_len 

[rank3]:     actual_seq_len = compute_actual_seq_len(position_ids) 

[rank3]:   File "/work/share/zjc/ms-llm/MindSpeed-LLM/mindspeed_llm/training/utils.py", line 51, in compute_actual_seq_len 

[rank3]:     zero_pos = (seq == 0).nonzero()[1:].squeeze(dim=1) 

[rank3]: RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnFlashAttentionVarLenScore. 

[rank3]: Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1. 

[rank3]: [ERROR] 2025-04-18-11:04:30 (PID:1525409, Device:3, RankID:3) ERR00100 PTA call acl api failed