镜像版本：mindie:dev-2.0.T18.B010-800I-A2-py311-ubuntu22.04-aarch64
芯片：310P
日志打印：
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
make: Nothing to be done for 'default'.
make: Leaving directory '/l00613958/MindSpeed-MM-master/megatron/core/datasets'
>>> done with dataset index builder. Compilation time: 0.630 seconds
building VLMModel ...
image encoder pipeline config: pp_rank:0, pre_process:True, post_process:True, local_num_layers:32
text decoder pipeline config: pp_rank:1, pre_process:False, post_process:False, local_num_layers:10
text decoder pipeline config: pp_rank:3, pre_process:False, post_process:True, local_num_layers:7
.....text decoder pipeline config: pp_rank:0, pre_process:True, post_process:False, local_num_layers:1
.. > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 1453814272
. > number of parameters on (tensor, pipeline) model parallel rank (0, 2): 2330577920
> number of parameters on (tensor, pipeline) model parallel rank (0, 3): 2176405504
> number of parameters on (tensor, pipeline) model parallel rank (0, 1): 2330577920
loading release checkpoint from ckpt/mm_path/Qwen2-VL-7B-Instruct
could not find arguments in the checkpoint ...
Traceback (most recent call last):
File "/l00613958/MindSpeed-MM-master/inference_vlm.py", line 29, in <module>
main()
File "/l00613958/MindSpeed-MM-master/inference_vlm.py", line 25, in main
vlm_pipeline_dict[inference_config.pipeline_class](inference_config)()
File "/l00613958/MindSpeed-MM-master/mindspeed_mm/tasks/inference/pipeline/qwen2vl_pipeline.py", line 23, in __init__
self.model = ParallelWrapper(model_provider)
File "/l00613958/MindSpeed-MM-master/mindspeed_mm/tasks/inference/pipeline/parallel_wrapper.py", line 38, in __init__
load_checkpoint(model, None, None, 'load')
File "/l00613958/MindSpeed-MM-master/megatron/training/checkpointing.py", line 839, in load_checkpoint
model[0].load_state_dict(state_dict['model'], strict=strict)
File "/root/miniconda3/envs/test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2152, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for VLMModel:
While copying the parameter named "image_encoder.encoder.patch_embed.proj.weight", whose dimensions in the model are torch.Size([1280, 3, 2, 14, 14]) and whose dimensions in the checkpoint are torch.Size([1280, 3, 2, 14, 14]), an exception occurred : ('InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:200 OPS function error: Identity, error code is 500002\n[ERROR] 2025-05-24-08:19:00 (PID:26408, Device:0, RankID:0) ERR01100 OPS call acl api failed\n[Error]: A GE error occurs in the system.\n Rectify the fault based on the error information in the ascend log.\nEZ3002: [PID: 26408] 2025-05-24-08:19:00.365.148 Optype [TransData] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[Dynamic shape check]: The format and dtype is not precisely equivalent to format and dtype in op information library[Static shape check]:The format and dtype is not precisely equivalent to format and dtype in op information library.\n Possible Cause: The operator type is unsupported in the operator information library due to specification mismatch.\n Solution: Submit an issue to request for support at https://gitee.com/ascend, or remove this type of operators from your model.\n TraceBack (most recent call last):\n Optype [TransData] of Ops kernel [aicpu_ascend_kernel] is unsupported. Reason: Transdata op, groups should be greater than 1, but now is 1.\n No supported Ops kernel and engine are found for [trans_TransData_79], optype [TransData].\n Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed[FUNC:operator()][FILE:engine_place.cc][LINE:148]\n RunAllSubgraphs failed, graph=online.[FUNC:RunAllSubgraphs][FILE:engine_place.cc][LINE:122]\n build graph failed, graph id:33, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1623]\n [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]\n [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]\n build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]\n',).
...Traceback (most recent call last):
File "/l00613958/MindSpeed-MM-master/inference_vlm.py", line 29, in <module>
main()
File "/l00613958/MindSpeed-MM-master/inference_vlm.py", line 25, in main
vlm_pipeline_dict[inference_config.pipeline_class](inference_config)()
File "/l00613958/MindSpeed-MM-master/mindspeed_mm/tasks/inference/pipeline/qwen2vl_pipeline.py", line 23, in __init__
self.model = ParallelWrapper(model_provider)
File "/l00613958/MindSpeed-MM-master/mindspeed_mm/tasks/inference/pipeline/parallel_wrapper.py", line 38, in __init__
load_checkpoint(model, None, None, 'load')
File "/l00613958/MindSpeed-MM-master/megatron/training/checkpointing.py", line 922, in load_checkpoint
torch.distributed.barrier()
File "/root/miniconda3/envs/test/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
return func(*args, **kwargs)
File "/root/miniconda3/envs/test/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
work = default_pg.barrier(opts=opts)
RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:160 OPS function error: HcclAllreduce, error code is 6
[ERROR] 2025-05-24-08:19:02 (PID:26409, Device:1, RankID:1) ERR01100 OPS call acl api failed.
EI0009: [PID: 26409] 2025-05-24-08:19:01.939.525 Transport init error. Reason: [Create][DestLink]Create Dest error! createLink para:rank[1]-localUserrank[1]-localIpAddr[192.168.1.100], dst_rank[0]-remoteUserrank[0]-remote_ip_addr[192.168.1.100]
Solution: Check other NPUs that are not reporting errors, or check if there are any abnormalities on the other end's NPU.
点赞
收藏
回复
分享
举报
已解决
镜像版本：mindie:dev-2.0.T18.B010-800I-A2-py311-ubuntu22.04-aarch64
芯片：310P
日志打印：
WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it
make: Nothing to be done for 'default'.
make: Leaving directory '/l00613958/MindSpeed-MM-master/megatron/core/datasets'
>>> done with dataset index builder. Compilation time: 0.630 seconds
building VLMModel ...
image encoder pipeline config: pp_rank:0, pre_process:True, post_process:True, local_num_layers:32
text decoder pipeline config: pp_rank:1, pre_process:False, post_process:False, local_num_layers:10
text decoder pipeline config: pp_rank:3, pre_process:False, post_process:True, local_num_layers:7
.....text decoder pipeline config: pp_rank:0, pre_process:True, post_process:False, local_num_layers:1
.. > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 1453814272
. > number of parameters on (tensor, pipeline) model parallel rank (0, 2): 2330577920
> number of parameters on (tensor, pipeline) model parallel rank (0, 3): 2176405504
> number of parameters on (tensor, pipeline) model parallel rank (0, 1): 2330577920
loading release checkpoint from ckpt/mm_path/Qwen2-VL-7B-Instruct
could not find arguments in the checkpoint ...
Traceback (most recent call last):
File "/l00613958/MindSpeed-MM-master/inference_vlm.py", line 29, in <module>
main()
File "/l00613958/MindSpeed-MM-master/inference_vlm.py", line 25, in main
vlm_pipeline_dict[inference_config.pipeline_class](inference_config)()
File "/l00613958/MindSpeed-MM-master/mindspeed_mm/tasks/inference/pipeline/qwen2vl_pipeline.py", line 23, in __init__
self.model = ParallelWrapper(model_provider)
File "/l00613958/MindSpeed-MM-master/mindspeed_mm/tasks/inference/pipeline/parallel_wrapper.py", line 38, in __init__
load_checkpoint(model, None, None, 'load')
File "/l00613958/MindSpeed-MM-master/megatron/training/checkpointing.py", line 839, in load_checkpoint
model[0].load_state_dict(state_dict['model'], strict=strict)
File "/root/miniconda3/envs/test/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2152, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for VLMModel:
While copying the parameter named "image_encoder.encoder.patch_embed.proj.weight", whose dimensions in the model are torch.Size([1280, 3, 2, 14, 14]) and whose dimensions in the checkpoint are torch.Size([1280, 3, 2, 14, 14]), an exception occurred : ('InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:200 OPS function error: Identity, error code is 500002\n[ERROR] 2025-05-24-08:19:00 (PID:26408, Device:0, RankID:0) ERR01100 OPS call acl api failed\n[Error]: A GE error occurs in the system.\n Rectify the fault based on the error information in the ascend log.\nEZ3002: [PID: 26408] 2025-05-24-08:19:00.365.148 Optype [TransData] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[Dynamic shape check]: The format and dtype is not precisely equivalent to format and dtype in op information library[Static shape check]:The format and dtype is not precisely equivalent to format and dtype in op information library.\n Possible Cause: The operator type is unsupported in the operator information library due to specification mismatch.\n Solution: Submit an issue to request for support at https://gitee.com/ascend, or remove this type of operators from your model.\n TraceBack (most recent call last):\n Optype [TransData] of Ops kernel [aicpu_ascend_kernel] is unsupported. Reason: Transdata op, groups should be greater than 1, but now is 1.\n No supported Ops kernel and engine are found for [trans_TransData_79], optype [TransData].\n Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed[FUNC:operator()][FILE:engine_place.cc][LINE:148]\n RunAllSubgraphs failed, graph=online.[FUNC:RunAllSubgraphs][FILE:engine_place.cc][LINE:122]\n build graph failed, graph id:33, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1623]\n [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]\n [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]\n build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]\n',).
...Traceback (most recent call last):
File "/l00613958/MindSpeed-MM-master/inference_vlm.py", line 29, in <module>
main()
File "/l00613958/MindSpeed-MM-master/inference_vlm.py", line 25, in main
vlm_pipeline_dict[inference_config.pipeline_class](inference_config)()
File "/l00613958/MindSpeed-MM-master/mindspeed_mm/tasks/inference/pipeline/qwen2vl_pipeline.py", line 23, in __init__
self.model = ParallelWrapper(model_provider)
File "/l00613958/MindSpeed-MM-master/mindspeed_mm/tasks/inference/pipeline/parallel_wrapper.py", line 38, in __init__
load_checkpoint(model, None, None, 'load')
File "/l00613958/MindSpeed-MM-master/megatron/training/checkpointing.py", line 922, in load_checkpoint
torch.distributed.barrier()
File "/root/miniconda3/envs/test/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
return func(*args, **kwargs)
File "/root/miniconda3/envs/test/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
work = default_pg.barrier(opts=opts)
RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:160 OPS function error: HcclAllreduce, error code is 6
[ERROR] 2025-05-24-08:19:02 (PID:26409, Device:1, RankID:1) ERR01100 OPS call acl api failed.
EI0009: [PID: 26409] 2025-05-24-08:19:01.939.525 Transport init error. Reason: [Create][DestLink]Create Dest error! createLink para:rank[1]-localUserrank[1]-localIpAddr[192.168.1.100], dst_rank[0]-remoteUserrank[0]-remote_ip_addr[192.168.1.100]
Solution: Check other NPUs that are not reporting errors, or check if there are any abnormalities on the other end's NPU.