2024-03-20 07:36:26,442	ERROR train.py:349 in <module> -- Raise exception from om-a09mw9zj-worker-0 with rank id: 0
Traceback (most recent call last):
  File "/data/zhaoguochun/InternLM/train.py", line 347, in <module>
    main(args)
  File "/data/zhaoguochun/InternLM/train.py", line 255, in main
    _, _, loss = trainer.execute_schedule(
  File "/data/zhaoguochun/InternLM/internlm/core/trainer.py", line 213, in execute_schedule
    return self._schedule.forward_backward_step(self._engine, data_iter, **kwargs)
  File "/data/zhaoguochun/InternLM/internlm/utils/timeout.py", line 102, in wrapper
    result = func(*args, **kwargs)
  File "/data/zhaoguochun/InternLM/internlm/core/scheduler/no_pipeline_scheduler.py", line 193, in forward_backward_step
    _output, _loss, _moe_loss = self._train_one_batch(
  File "/data/zhaoguochun/InternLM/internlm/core/scheduler/no_pipeline_scheduler.py", line 135, in _train_one_batch
    engine.backward(loss)
  File "/data/zhaoguochun/InternLM/internlm/core/engine.py", line 145, in backward
    return self.optimizer.backward(loss)
  File "/data/zhaoguochun/InternLM/internlm/solver/optimizer/hybrid_zero_optim.py", line 515, in backward
    loss.backward(retain_graph=retain_graph)
  File "/opt/miniconda3/envs/dipu_dev_py39/lib/python3.9/site-packages/torch/_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "/opt/miniconda3/envs/dipu_dev_py39/lib/python3.9/site-packages/torch/autograd/__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: ascend device error, expr = ::aclrtSynchronizeStream(stream), ret = 507035, error msg = EZ9999: Inner Error!
EZ9999  The error from device(chipId:0, dieId:0), serial number is 6, there is an aivec error exception, core id is 8, error code = 0x800000, dump info: pc start: 0x1240c09f6000, current: 0x1240c09f6118, vec error info: 0xa01794b550, mte error info: 0xd803001e37, ifu error info: 0x7b0ffac1fde00, ccu error info: 0x18b42b9839030b04, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x124100fffc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1164]
        TraceBack (most recent call last):

....
        The error from device(chipId:0, dieId:0), serial number is 6, there is an aivec error exception, core id is 7, error code = 0x800000, dump info: pc start: 0x1240c09f6000, current: 0x1240c09f6118, vec error info: 0x6900bb2bb8, mte error info: 0xd803001e37, ifu error info: 0x44105736ed80, ccu error info: 0x24c14d0260f08312, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x124100fffc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1164]
        The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x3001e37, fixp_error1 info: 0xd8 fsmId:1, tslot:7, thread:0, ctxid:0, blk:47, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1176]
        Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:task_info.cc][LINE:1677]
        AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1454]
        Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=51437, flip_num=474, fault kernel_name=NLLLossGrad_90c5e73ef45571bf8d070cf0946de930_high_performance__kernel0, program id=95, hash=15318471335765398531.[FUNC:GetError][FILE:stream.cc][LINE:1454]
        [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1454]
        rtStreamSynchronize execute failed, reason=[vector core exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
        synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

Exception raised from syncStream at /data/zhaoguochun/deeplink.framework/dipu/torch_dipu/csrc_dipu/vendor/ascend/deviceimpl.cpp:207 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f09a8ae5d77 in /opt/miniconda3/envs/dipu_dev_py39/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f09a8aafabb in /opt/miniconda3/envs/dipu_dev_py39/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #2: dipu::devapis::syncStream(void*) + 0x424 (0x7f09a0bc4614 in /data/zhaoguochun/deeplink.framework/dipu/torch_dipu/libtorch_dipu.so)
frame #3: dipu::DIPUStream::synchronize() const + 0x67 (0x7f09a0b7e127 in /data/zhaoguochun/deeplink.framework/dipu/torch_dipu/libtorch_dipu.so)
frame #4: dipu::native::synchronizeIfEnable() + 0x5e (0x7f09a0b7e26e in /data/zhaoguochun/deeplink.framework/dipu/torch_dipu/libtorch_dipu.so)