[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:14.758.780 [mindspore/ccsrc/frontend/parallel/auto_parallel/graph_costmodel.cc:137] CheckVisitedEdgeConsistency] Inconsistency occurred at edge: AddInfo24282428-ReshapeInfo24272427
[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:14.759.362 [mindspore/ccsrc/frontend/parallel/ops_info/reshape_info.cc:821] CheckStrategyConsistencyByInputLayout] ReshapeInfo24632463's desired input layout is:
device arrangement = [ 8 ]
tensor map = [ -1 -1 ]
tensor shape = [ 1024 5120 ]
device arrangement origin = [ 1 1 8 ]
tensor map origin = [ 2 1 ]
tensor shape origin = [ 1024 5120 ], while the selected input layout is:
device arrangement = [ 8 ]
tensor map = [ -1 0 ]
tensor shape = [ 1024 5120 ]
device arrangement origin = [ 1 8 ]
tensor map origin = [ 1 0 ]
tensor shape origin = [ 1024 5120 ] and the output layout is:
device arrangement = [ 8 ]
tensor map = [ -1 -1 0 -1 ]
tensor shape = [ 1 1024 40 128 ]
device arrangement origin = [ 1 1 8 1 ]
tensor map origin = [ 3 2 1 0 ]
tensor shape origin = [ 1 1024 40 128 ]
[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:14.759.393 [mindspore/ccsrc/frontend/parallel/auto_parallel/graph_costmodel.cc:137] CheckVisitedEdgeConsistency] Inconsistency occurred at edge: AddInfo24642464-ReshapeInfo24632463
[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:16.103.772 [mindspore/ccsrc/frontend/parallel/auto_parallel/edge_costmodel.cc:377] GetNextOpStrategyByPrevOpStrategyWithMiniComm] Inconsistency occurred at edge: CastInfo77-VirtualOutputInfo66
[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:16.104.597 [mindspore/ccsrc/frontend/parallel/auto_parallel/edge_costmodel.cc:568] CheckStrategyConsistency] There are redistribution cost occurs at edge: CastInfo77-VirtualOutputInfo66.
[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:16.104.633 [mindspore/ccsrc/frontend/parallel/auto_parallel/graph_costmodel.cc:143] CheckVisitedEdgeConsistency] Inconsistency occurred at edge: CastInfo77-VirtualOutputInfo66
Traceback (most recent call last):
File "/data/shaohongming/Qwen/mindformers/research/qwen/run_qwen.py", line 169, in <module>
main(task=args.task,
File "/data/shaohongming/Qwen/mindformers/research/qwen/run_qwen.py", line 113, in main
result = task.predict(input_data=prompt,
File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/_checkparam.py", line 1313, in wrapper
return func(*args, **kwargs)
File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/trainer.py", line 640, in predict
output_result = self.trainer.predict(
File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 316, in predict
return self.predict_process(config=config,
File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/base_trainer.py", line 917, in predict_process
transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True)
File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/utils.py", line 319, in transform_and_load_checkpoint
build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict)
File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/utils.py", line 441, in build_model
model.infer_predict_layout(dataset)
File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/train/model.py", line 1882, in infer_predict_layout
predict_net.compile(*predict_data)
File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/nn/cell.py", line 997, in compile
_cell_graph_executor.compile(self, phase=self.phase,
File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/common/api.py", line 1547, in compile
result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
ValueError: The accumulate of x_shape must be equal to out_shape, but got x_shape: [const vector]{128, 5120}, and out_shape: [const vector]{1, 1024, 5120}
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/core/ops/reshape.cc:89 update_shape
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
# 0 In file /data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/modules/layers.py:501
output = self.reshape(x, out_shape)
^
(See file '/data/shaohongming/Qwen/mindformers/research/rank_7/om/analyze_fail.ir' for more details. Get instructions about `analyze_fail.ir` at https://www.mindspore.cn/search?inputValue=analyze_fail.ir)
1.错误日志
./output/log/rank_7 其余log报错信息一致
[WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:14.758.780 [mindspore/ccsrc/frontend/parallel/auto_parallel/graph_costmodel.cc:137] CheckVisitedEdgeConsistency] Inconsistency occurred at edge: AddInfo24282428-ReshapeInfo24272427 [WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:14.759.362 [mindspore/ccsrc/frontend/parallel/ops_info/reshape_info.cc:821] CheckStrategyConsistencyByInputLayout] ReshapeInfo24632463's desired input layout is: device arrangement = [ 8 ] tensor map = [ -1 -1 ] tensor shape = [ 1024 5120 ] device arrangement origin = [ 1 1 8 ] tensor map origin = [ 2 1 ] tensor shape origin = [ 1024 5120 ], while the selected input layout is: device arrangement = [ 8 ] tensor map = [ -1 0 ] tensor shape = [ 1024 5120 ] device arrangement origin = [ 1 8 ] tensor map origin = [ 1 0 ] tensor shape origin = [ 1024 5120 ] and the output layout is: device arrangement = [ 8 ] tensor map = [ -1 -1 0 -1 ] tensor shape = [ 1 1024 40 128 ] device arrangement origin = [ 1 1 8 1 ] tensor map origin = [ 3 2 1 0 ] tensor shape origin = [ 1 1024 40 128 ] [WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:14.759.393 [mindspore/ccsrc/frontend/parallel/auto_parallel/graph_costmodel.cc:137] CheckVisitedEdgeConsistency] Inconsistency occurred at edge: AddInfo24642464-ReshapeInfo24632463 [WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:16.103.772 [mindspore/ccsrc/frontend/parallel/auto_parallel/edge_costmodel.cc:377] GetNextOpStrategyByPrevOpStrategyWithMiniComm] Inconsistency occurred at edge: CastInfo77-VirtualOutputInfo66 [WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:16.104.597 [mindspore/ccsrc/frontend/parallel/auto_parallel/edge_costmodel.cc:568] CheckStrategyConsistency] There are redistribution cost occurs at edge: CastInfo77-VirtualOutputInfo66. [WARNING] PARALLEL(3797260,fffc90465510,python):2024-03-15-20:19:16.104.633 [mindspore/ccsrc/frontend/parallel/auto_parallel/graph_costmodel.cc:143] CheckVisitedEdgeConsistency] Inconsistency occurred at edge: CastInfo77-VirtualOutputInfo66 Traceback (most recent call last): File "/data/shaohongming/Qwen/mindformers/research/qwen/run_qwen.py", line 169, in <module> main(task=args.task, File "/data/shaohongming/Qwen/mindformers/research/qwen/run_qwen.py", line 113, in main result = task.predict(input_data=prompt, File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/_checkparam.py", line 1313, in wrapper return func(*args, **kwargs) File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/trainer.py", line 640, in predict output_result = self.trainer.predict( File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 316, in predict return self.predict_process(config=config, File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/base_trainer.py", line 917, in predict_process transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True) File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/utils.py", line 319, in transform_and_load_checkpoint build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict) File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/trainer/utils.py", line 441, in build_model model.infer_predict_layout(dataset) File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/train/model.py", line 1882, in infer_predict_layout predict_net.compile(*predict_data) File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/nn/cell.py", line 997, in compile _cell_graph_executor.compile(self, phase=self.phase, File "/data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindspore/common/api.py", line 1547, in compile result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode()) ValueError: The accumulate of x_shape must be equal to out_shape, but got x_shape: [const vector]{128, 5120}, and out_shape: [const vector]{1, 1024, 5120} ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/core/ops/reshape.cc:89 update_shape ---------------------------------------------------- - The Traceback of Net Construct Code: ---------------------------------------------------- # 0 In file /data/apps/tools/miniconda3/envs/shm_qwen/lib/python3.8/site-packages/mindformers/modules/layers.py:501 output = self.reshape(x, out_shape) ^ (See file '/data/shaohongming/Qwen/mindformers/research/rank_7/om/analyze_fail.ir' for more details. Get instructions about `analyze_fail.ir` at https://www.mindspore.cn/search?inputValue=analyze_fail.ir)mindspore 2.2.11 mindformer dev 机器A800(9000)NPU 910proB