[WARNING] DEVICE(139206,fffc8ef6f1f0,python):2023-08-21-14:45:11.241.993 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:743] GetDumpPath] The environment variable 'MS_OM_PATH' is not set, the files of node dump will save to the process local path, as ./rank_id/node_dump/... 

[ERROR] DEVICE(139206,fffc8ef6f1f0,python):2023-08-21-14:45:11.243.334 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:761] DumpTaskExceptionInfo] Task fail infos task_id: 2, stream_id: 581, tid: 139206, device_id: 0, retcode: 507011 ( model execute failed) 

[WARNING] DEVICE(139206,fffbaf7ef1f0,python):2023-08-21-14:45:11.324.752 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_data_queue.cc:282] Push] Device queue thread had been interrupted by TdtHandle::DestroyHandle, you can ignore the above error: 'failed to send...'. In this scenario, the training ends first without using all epoch(s) data, and the data preprocessing is blocked by the data transmission channel on the device side. So we force the data transmission channel to stop. 

[WARNING] MD(139206,fffbaf7ef1f0,python):2023-08-21-14:45:11.324.999 [mindspore/ccsrc/minddata/dataset/engine/datasetops/data_queue_op.cc:243] SendDataToAscend] Thread has already been terminated. 

2023-08-21 14:45:11,346 - mindformers - ERROR - Traceback (most recent call last): 

  File "/home/nfs/appnfs/aixczbm/data/mindformers-dev/scripts/mf_parallel0/mindformers/tools/cloud_adapter/cloud_monitor.py", line 33, in wrapper 

    result = run_func(*args, **kwargs) 

  File "run_mindformer.py", line 99, in main 

    trainer.train(config, is_full_config=True) 

  File "/home/nfs/appnfs/aixczbm/data/mindformers-dev/scripts/mf_parallel0/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 98, in train 

    **kwargs) 

  File "/home/nfs/appnfs/aixczbm/data/mindformers-dev/scripts/mf_parallel0/mindformers/trainer/base_trainer.py", line 624, in training_process 

    initial_epoch=config.runner_config.initial_epoch) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 1051, in train 

    initial_epoch=initial_epoch) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 98, in wrapper 

    func(self, *args, **kwargs) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 625, in _train 

    cb_params, sink_size, initial_epoch, valid_infos) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 703, in _train_dataset_sink_process 

    outputs = train_network(*inputs) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 618, in __call__ 

    out = self.compile_and_run(*args) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 1007, in compile_and_run 

    return _cell_graph_executor(self, *new_inputs, phase=self.phase) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/common/api.py", line 1192, in __call__ 

    return self.run(obj, *args, phase=phase) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/common/api.py", line 1229, in run 

    return self._exec_pip(obj, *args, phase=phase_real) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper 

    results = fn(*arg, **kwargs) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/common/api.py", line 1211, in _exec_pip 

    return self._graph_executor(args, phase) 

RuntimeError: Run task error! 

 

---------------------------------------------------- 

- Ascend Error Message: 

---------------------------------------------------- 

EE9999: Inner Error, Please contact support engineer! 

EE9999  Task execute failed, base info: device_id=0, stream_id=581, task_id=2, flip_num=0, task_type=3.[FUNC:GetError][FILE:stream.cc][LINE:1044] 

        TraceBack (most recent call last): 

        rtStreamSynchronize execute failed, reason=[the model stream execute failed][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] 

 

---------------------------------------------------- 

- C++ Call Stack: (For framework developers) 

---------------------------------------------------- 

mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_graph_executor.cc:214 RunGraph 

 

Traceback (most recent call last): 

  File "run_mindformer.py", line 282, in <module> 

    main(config_) 

  File "/home/nfs/appnfs/aixczbm/data/mindformers-dev/scripts/mf_parallel0/mindformers/tools/cloud_adapter/cloud_monitor.py", line 37, in wrapper 

    raise exc 

  File "/home/nfs/appnfs/aixczbm/data/mindformers-dev/scripts/mf_parallel0/mindformers/tools/cloud_adapter/cloud_monitor.py", line 33, in wrapper 

    result = run_func(*args, **kwargs) 

  File "run_mindformer.py", line 99, in main 

    trainer.train(config, is_full_config=True) 

  File "/home/nfs/appnfs/aixczbm/data/mindformers-dev/scripts/mf_parallel0/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 98, in train 

    **kwargs) 

  File "/home/nfs/appnfs/aixczbm/data/mindformers-dev/scripts/mf_parallel0/mindformers/trainer/base_trainer.py", line 624, in training_process 

    initial_epoch=config.runner_config.initial_epoch) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 1051, in train 

    initial_epoch=initial_epoch) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 98, in wrapper 

    func(self, *args, **kwargs) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 625, in _train 

    cb_params, sink_size, initial_epoch, valid_infos) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 703, in _train_dataset_sink_process 

    outputs = train_network(*inputs) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 618, in __call__ 

    out = self.compile_and_run(*args) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 1007, in compile_and_run 

    return _cell_graph_executor(self, *new_inputs, phase=self.phase) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/common/api.py", line 1192, in __call__ 

    return self.run(obj, *args, phase=phase) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/common/api.py", line 1229, in run 

    return self._exec_pip(obj, *args, phase=phase_real) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper 

    results = fn(*arg, **kwargs) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/common/api.py", line 1211, in _exec_pip 

    return self._graph_executor(args, phase) 

RuntimeError: Run task error! 

 

---------------------------------------------------- 

- Ascend Error Message: 

---------------------------------------------------- 

EE9999: Inner Error, Please contact support engineer! 

EE9999  Task execute failed, base info: device_id=0, stream_id=581, task_id=2, flip_num=0, task_type=3.[FUNC:GetError][FILE:stream.cc][LINE:1044] 

        TraceBack (most recent call last): 

        rtStreamSynchronize execute failed, reason=[the model stream execute failed][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49] 

 

---------------------------------------------------- 

- C++ Call Stack: (For framework developers) 

---------------------------------------------------- 

mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_graph_executor.cc:214 RunGraph