使用Mindspore多卡推理Qwen14B出错
收藏回复举报
使用Mindspore多卡推理Qwen14B出错
t('forum.solved') 已解决
发表于2024-03-15 20:31:18
0 查看

1.错误日志

./output/log/rank_7 其余log报错信息一致

[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:14.758.780 [mindspore/ccsrc/frontend/parallel/auto_parallel/graph_costmodel.cc:137] CheckVisitedEdgeConsistency] Inconsistency occurred at edge: AddInfo24282428-ReshapeInfo24272427
[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:14.759.362 [mindspore/ccsrc/frontend/parallel/ops_info/reshape_info.cc:821] CheckStrategyConsistencyByInputLayout] ReshapeInfo24632463's desired input layout is:
device arrangement = [ 8 ]
tensor map = [ -1 -1 ]
tensor shape = [ 1024 5120 ]
device arrangement origin = [ 1 1 8 ]
tensor map origin = [ 2 1 ]
tensor shape origin = [ 1024 5120 ], while the selected input layout is: 
device arrangement = [ 8 ]
tensor map = [ -1 0 ]
tensor shape = [ 1024 5120 ]
device arrangement origin = [ 1 8 ]
tensor map origin = [ 1 0 ]
tensor shape origin = [ 1024 5120 ] and the output layout is: 
device arrangement = [ 8 ]
tensor map = [ -1 -1 0 -1 ]
tensor shape = [ 1 1024 40 128 ]
device arrangement origin = [ 1 1 8 1 ]
tensor map origin = [ 3 2 1 0 ]
tensor shape origin = [ 1 1024 40 128 ]
[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:14.759.393 [mindspore/ccsrc/frontend/parallel/auto_parallel/graph_costmodel.cc:137] CheckVisitedEdgeConsistency] Inconsistency occurred at edge: AddInfo24642464-ReshapeInfo24632463
[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:16.103.772 [mindspore/ccsrc/frontend/parallel/auto_parallel/edge_costmodel.cc:377] GetNextOpStrategyByPrevOpStrategyWithMiniComm] Inconsistency occurred at edge: CastInfo77-VirtualOutputInfo66
[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:16.104.597 [mindspore/ccsrc/frontend/parallel/auto_parallel/edge_costmodel.cc:568] CheckStrategyConsistency] There are redistribution cost occurs at edge: CastInfo77-VirtualOutputInfo66.
[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:16.104.633 [mindspore/ccsrc/frontend/parallel/auto_parallel/graph_costmodel.cc:143] CheckVisitedEdgeConsistency] Inconsistency occurred at edge: CastInfo77-VirtualOutputInfo66
Traceback (most recent call last):
  File "/data/shaohongming/Qwen/mindformers/research/qwen/run_qwen.py", line 169, in <module>
    main(task=args.task,
  File "/data/shaohongming/Qwen/mindformers/research/qwen/run_qwen.py", line 113, in main
    result = task.predict(input_data=prompt,
  File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/_checkparam.py", line 1313, in wrapper
    return func(*args, **kwargs)
  File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/trainer.py", line 640, in predict
    output_result = self.trainer.predict(
  File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 316, in predict
    return self.predict_process(config=config,
  File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/base_trainer.py", line 917, in predict_process
    transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True)
  File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/utils.py", line 319, in transform_and_load_checkpoint
    build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict)
  File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/utils.py", line 441, in build_model
    model.infer_predict_layout(dataset)
  File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/train/model.py", line 1882, in infer_predict_layout
    predict_net.compile(*predict_data)
  File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/nn/cell.py", line 997, in compile
    _cell_graph_executor.compile(self, phase=self.phase,
  File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/common/api.py", line 1547, in compile
    result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
ValueError: The accumulate of x_shape must be equal to out_shape, but got x_shape: [const vector]{128, 5120}, and out_shape: [const vector]{1, 1024, 5120}

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/core/ops/reshape.cc:89 update_shape

----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
# 0 In file /data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/modules/layers.py:501
        output = self.reshape(x, out_shape)
                 ^
 (See file '/data/shaohongming/Qwen/mindformers/research/rank_7/om/analyze_fail.ir' for more details. Get instructions about `analyze_fail.ir` at https://www.mindspore.cn/search?inputValue=analyze_fail.ir)

mindspore 2.2.11 mindformer dev 机器A800(9000)NPU 910proB

我要发帖子