ubuntu20.04 Atlas 300V Pro Video python版本3.10.13 torch-npu2.1.0 torch2.1.0 llama-factory，0.9.1 glm-4-9b-chat ，、
[INFO|2024-12-20 01:34:51] llamafactory.model.model_utils.attention:157 >> Using vanilla attention implementation.
[INFO|2024-12-20 01:34:51] llamafactory.model.adapter:157 >> Upcasting trainable params to float32.
[INFO|2024-12-20 01:34:51] llamafactory.model.adapter:157 >> Fine-tuning method: LoRA
[INFO|2024-12-20 01:34:51] llamafactory.model.model_utils.misc:157 >> Found linear modules: dense,dense_h_to_4h,dense_4h_to_h,query_key_value
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:37<00:00, 3.74s/it]
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:37<00:00, 3.74s/it]
[INFO|2024-12-20 01:34:52] llamafactory.model.loader:157 >> trainable params: 21,176,320 || all params: 9,421,127,680 || trainable%: 0.2248
/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/trainer.py:54: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `CustomSeq2SeqTrainer.__init__`. Use `processing_class` instead.
super().__init__(**kwargs)
Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
[INFO|trainer.py:698] 2024-12-20 01:34:52,550 >> Using auto half precision backend
/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/trainer.py:54: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `CustomSeq2SeqTrainer.__init__`. Use `processing_class` instead.
super().__init__(**kwargs)
/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/trainer.py:54: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `CustomSeq2SeqTrainer.__init__`. Use `processing_class` instead.
super().__init__(**kwargs)
[INFO|trainer.py:2313] 2024-12-20 01:34:53,309 >> ***** Running training *****
[INFO|trainer.py:2314] 2024-12-20 01:34:53,309 >> Num examples = 231
[INFO|trainer.py:2315] 2024-12-20 01:34:53,309 >> Num Epochs = 10
[INFO|trainer.py:2316] 2024-12-20 01:34:53,309 >> Instantaneous batch size per device = 2
[INFO|trainer.py:2319] 2024-12-20 01:34:53,309 >> Total train batch size (w. parallel, distributed & accumulation) = 64
[INFO|trainer.py:2320] 2024-12-20 01:34:53,309 >> Gradient Accumulation steps = 8
[INFO|trainer.py:2321] 2024-12-20 01:34:53,310 >> Total optimization steps = 30
[INFO|trainer.py:2322] 2024-12-20 01:34:53,317 >> Number of trainable parameters = 21,176,320
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:40<00:00, 4.01s/it]
/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/trainer.py:54: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `CustomSeq2SeqTrainer.__init__`. Use `processing_class` instead.
super().__init__(**kwargs)
.[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 823032] 2024-12-20-01:35:17.702.508 Op NLLLossGrad does not has any binary.
TraceBack (most recent call last):
Kernel Run failed. opType: 41, NLLLossGrad
launch failed for NLLLossGrad, errno:561000.
[ERROR] 2024-12-20-01:35:17 (PID:823032, Device:0, RankID:0) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff88294898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff8824d2a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff7d28b004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff7d5dd840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff7cd20a70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff7cd20e98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff7cd1ed70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff882bb6ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffff91cee624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffff91b7262c in /lib/aarch64-linux-gnu/libc.so.6)
Traceback (most recent call last):
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
launch()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
run_exp()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
return inner_training_loop(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
self.accelerator.backward(loss, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
self.scaler.scale(loss).backward(**kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1071, in unpack_hook
frame.recompute_fn(*args)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1194, in recompute_fn
fn(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 623, in forward
layernorm_output = self.input_layernorm(hidden_states)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 161, in forward
return (self.weight * hidden_states).to(input_dtype)
RuntimeError: The Inner error is reported as above.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-20-01:35:17 (PID:823032, Device:0, RankID:0) ERR00005 PTA internal error
..[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 823034] 2024-12-20-01:35:18.979.185 Op NLLLossGrad does not has any binary.
TraceBack (most recent call last):
Kernel Run failed. opType: 41, NLLLossGrad
launch failed for NLLLossGrad, errno:561000.
[ERROR] 2024-12-20-01:35:18 (PID:823034, Device:2, RankID:2) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff9592b898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff958e42a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff8a922004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff8ac74840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff8a3b7a70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff8a3b7e98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff8a3b5d70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff959526ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffff9f385624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffff9f20962c in /lib/aarch64-linux-gnu/libc.so.6)
Traceback (most recent call last):
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
launch()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
run_exp()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
return inner_training_loop(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
self.accelerator.backward(loss, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
self.scaler.scale(loss).backward(**kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 823035] 2024-12-20-01:35:18.986.819 Op NLLLossGrad does not has any binary.
TraceBack (most recent call last):
Kernel Run failed. opType: 41, NLLLossGrad
launch failed for NLLLossGrad, errno:561000.
[ERROR] 2024-12-20-01:35:18 (PID:823035, Device:3, RankID:3) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff84f27898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff84ee02a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff79f1e004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff7a270840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff799b3a70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff799b3e98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff799b1d70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff84f4e6ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffff8e981624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffff8e80562c in /lib/aarch64-linux-gnu/libc.so.6)
torch.autograd.backward(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1071, in unpack_hook
frame.recompute_fn(*args)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1194, in recompute_fn
fn(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 623, in forward
layernorm_output = self.input_layernorm(hidden_states)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 158, in forward
variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
RuntimeError: The Inner error is reported as above.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-20-01:35:18 (PID:823034, Device:2, RankID:2) ERR00005 PTA internal error
Traceback (most recent call last):
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
launch()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
run_exp()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
return inner_training_loop(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
self.accelerator.backward(loss, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
self.scaler.scale(loss).backward(**kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1071, in unpack_hook
frame.recompute_fn(*args)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1194, in recompute_fn
fn(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 646, in forward
mlp_output = self.mlp(layernorm_output)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 582, in forward
output = self.dense_4h_to_h(intermediate_parallel)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 556, in forward
result = result + lora_B(lora_A(dropout(x))) * scaling
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: The Inner error is reported as above.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-20-01:35:18 (PID:823035, Device:3, RankID:3) ERR00005 PTA internal error
.[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 823033] 2024-12-20-01:35:23.770.426 Op NLLLossGrad does not has any binary.
TraceBack (most recent call last):
Kernel Run failed. opType: 41, NLLLossGrad
launch failed for NLLLossGrad, errno:561000.
[ERROR] 2024-12-20-01:35:23 (PID:823033, Device:1, RankID:1) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff794c2898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff7947b2a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff6e4b6004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff6e808840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff6df4ba70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff6df4be98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff6df49d70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff794e96ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffff82f1c624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffff82da062c in /lib/aarch64-linux-gnu/libc.so.6)
Traceback (most recent call last):
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
launch()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
run_exp()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
return inner_training_loop(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
self.accelerator.backward(loss, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
self.scaler.scale(loss).backward(**kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1071, in unpack_hook
frame.recompute_fn(*args)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1194, in recompute_fn
fn(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 625, in forward
attention_output, kv_cache = self.self_attention(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 450, in forward
mixed_x_layer = self.query_key_value(hidden_states)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 556, in forward
result = result + lora_B(lora_A(dropout(x))) * scaling
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: The Inner error is reported as above.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-20-01:35:23 (PID:823033, Device:1, RankID:1) ERR00005 PTA internal error
[2024-12-20 01:35:33,303] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 823032 closing signal SIGTERM
[2024-12-20 01:35:33,919] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 823033) of binary: /app/anaconda3/envs/python31013llama/bin/python
Traceback (most recent call last):
File "/app/anaconda3/envs/python31013llama/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2024-12-20_01:35:33
host : hbjj
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 823034)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2024-12-20_01:35:33
host : hbjj
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 823035)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-12-20_01:35:33
host : hbjj
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 823033)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Process ForkServerProcess-8:
Process ForkServerProcess-3:
Process ForkServerProcess-4:
Process ForkServerProcess-6:
Process ForkServerProcess-7:
Process ForkServerProcess-2:
Process ForkServerProcess-5:
Process ForkServerProcess-9:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "<string>", line 2, in get
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "<string>", line 2, in get
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "<string>", line 2, in get
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "<string>", line 2, in get
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
Traceback (most recent call last):
File "<string>", line 2, in get
Traceback (most recent call last):
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "<string>", line 2, in get
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
EOFError
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
EOFError
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "<string>", line 2, in get
File "<string>", line 2, in get
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
EOFError
/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
点赞
收藏
回复
分享
举报
已解决
ubuntu20.04 Atlas 300V Pro Video python版本3.10.13 torch-npu2.1.0 torch2.1.0 llama-factory，0.9.1 glm-4-9b-chat ，、
[INFO|2024-12-20 01:34:51] llamafactory.model.model_utils.attention:157 >> Using vanilla attention implementation.
[INFO|2024-12-20 01:34:51] llamafactory.model.adapter:157 >> Upcasting trainable params to float32.
[INFO|2024-12-20 01:34:51] llamafactory.model.adapter:157 >> Fine-tuning method: LoRA
[INFO|2024-12-20 01:34:51] llamafactory.model.model_utils.misc:157 >> Found linear modules: dense,dense_h_to_4h,dense_4h_to_h,query_key_value
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:37<00:00, 3.74s/it]
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:37<00:00, 3.74s/it]
[INFO|2024-12-20 01:34:52] llamafactory.model.loader:157 >> trainable params: 21,176,320 || all params: 9,421,127,680 || trainable%: 0.2248
/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/trainer.py:54: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `CustomSeq2SeqTrainer.__init__`. Use `processing_class` instead.
super().__init__(**kwargs)
Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
[INFO|trainer.py:698] 2024-12-20 01:34:52,550 >> Using auto half precision backend
/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/trainer.py:54: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `CustomSeq2SeqTrainer.__init__`. Use `processing_class` instead.
super().__init__(**kwargs)
/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/trainer.py:54: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `CustomSeq2SeqTrainer.__init__`. Use `processing_class` instead.
super().__init__(**kwargs)
[INFO|trainer.py:2313] 2024-12-20 01:34:53,309 >> ***** Running training *****
[INFO|trainer.py:2314] 2024-12-20 01:34:53,309 >> Num examples = 231
[INFO|trainer.py:2315] 2024-12-20 01:34:53,309 >> Num Epochs = 10
[INFO|trainer.py:2316] 2024-12-20 01:34:53,309 >> Instantaneous batch size per device = 2
[INFO|trainer.py:2319] 2024-12-20 01:34:53,309 >> Total train batch size (w. parallel, distributed & accumulation) = 64
[INFO|trainer.py:2320] 2024-12-20 01:34:53,309 >> Gradient Accumulation steps = 8
[INFO|trainer.py:2321] 2024-12-20 01:34:53,310 >> Total optimization steps = 30
[INFO|trainer.py:2322] 2024-12-20 01:34:53,317 >> Number of trainable parameters = 21,176,320
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:40<00:00, 4.01s/it]
/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/trainer.py:54: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `CustomSeq2SeqTrainer.__init__`. Use `processing_class` instead.
super().__init__(**kwargs)
.[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 823032] 2024-12-20-01:35:17.702.508 Op NLLLossGrad does not has any binary.
TraceBack (most recent call last):
Kernel Run failed. opType: 41, NLLLossGrad
launch failed for NLLLossGrad, errno:561000.
[ERROR] 2024-12-20-01:35:17 (PID:823032, Device:0, RankID:0) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff88294898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff8824d2a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff7d28b004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff7d5dd840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff7cd20a70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff7cd20e98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff7cd1ed70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff882bb6ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffff91cee624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffff91b7262c in /lib/aarch64-linux-gnu/libc.so.6)
Traceback (most recent call last):
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
launch()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
run_exp()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
return inner_training_loop(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
self.accelerator.backward(loss, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
self.scaler.scale(loss).backward(**kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1071, in unpack_hook
frame.recompute_fn(*args)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1194, in recompute_fn
fn(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 623, in forward
layernorm_output = self.input_layernorm(hidden_states)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 161, in forward
return (self.weight * hidden_states).to(input_dtype)
RuntimeError: The Inner error is reported as above.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-20-01:35:17 (PID:823032, Device:0, RankID:0) ERR00005 PTA internal error
..[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 823034] 2024-12-20-01:35:18.979.185 Op NLLLossGrad does not has any binary.
TraceBack (most recent call last):
Kernel Run failed. opType: 41, NLLLossGrad
launch failed for NLLLossGrad, errno:561000.
[ERROR] 2024-12-20-01:35:18 (PID:823034, Device:2, RankID:2) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff9592b898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff958e42a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff8a922004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff8ac74840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff8a3b7a70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff8a3b7e98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff8a3b5d70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff959526ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffff9f385624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffff9f20962c in /lib/aarch64-linux-gnu/libc.so.6)
Traceback (most recent call last):
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
launch()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
run_exp()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
return inner_training_loop(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
self.accelerator.backward(loss, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
self.scaler.scale(loss).backward(**kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 823035] 2024-12-20-01:35:18.986.819 Op NLLLossGrad does not has any binary.
TraceBack (most recent call last):
Kernel Run failed. opType: 41, NLLLossGrad
launch failed for NLLLossGrad, errno:561000.
[ERROR] 2024-12-20-01:35:18 (PID:823035, Device:3, RankID:3) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff84f27898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff84ee02a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff79f1e004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff7a270840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff799b3a70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff799b3e98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff799b1d70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff84f4e6ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffff8e981624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffff8e80562c in /lib/aarch64-linux-gnu/libc.so.6)
torch.autograd.backward(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1071, in unpack_hook
frame.recompute_fn(*args)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1194, in recompute_fn
fn(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 623, in forward
layernorm_output = self.input_layernorm(hidden_states)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 158, in forward
variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
RuntimeError: The Inner error is reported as above.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-20-01:35:18 (PID:823034, Device:2, RankID:2) ERR00005 PTA internal error
Traceback (most recent call last):
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
launch()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
run_exp()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
return inner_training_loop(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
self.accelerator.backward(loss, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
self.scaler.scale(loss).backward(**kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1071, in unpack_hook
frame.recompute_fn(*args)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1194, in recompute_fn
fn(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 646, in forward
mlp_output = self.mlp(layernorm_output)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 582, in forward
output = self.dense_4h_to_h(intermediate_parallel)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 556, in forward
result = result + lora_B(lora_A(dropout(x))) * scaling
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: The Inner error is reported as above.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-20-01:35:18 (PID:823035, Device:3, RankID:3) ERR00005 PTA internal error
.[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 823033] 2024-12-20-01:35:23.770.426 Op NLLLossGrad does not has any binary.
TraceBack (most recent call last):
Kernel Run failed. opType: 41, NLLLossGrad
launch failed for NLLLossGrad, errno:561000.
[ERROR] 2024-12-20-01:35:23 (PID:823033, Device:1, RankID:1) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff794c2898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff7947b2a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff6e4b6004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff6e808840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff6df4ba70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff6df4be98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff6df49d70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff794e96ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffff82f1c624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffff82da062c in /lib/aarch64-linux-gnu/libc.so.6)
Traceback (most recent call last):
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
launch()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
run_exp()
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
return inner_training_loop(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
self.accelerator.backward(loss, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
self.scaler.scale(loss).backward(**kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1071, in unpack_hook
frame.recompute_fn(*args)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1194, in recompute_fn
fn(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 625, in forward
attention_output, kv_cache = self.self_attention(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 450, in forward
mixed_x_layer = self.query_key_value(hidden_states)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 556, in forward
result = result + lora_B(lora_A(dropout(x))) * scaling
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: The Inner error is reported as above.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-20-01:35:23 (PID:823033, Device:1, RankID:1) ERR00005 PTA internal error
[2024-12-20 01:35:33,303] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 823032 closing signal SIGTERM
[2024-12-20 01:35:33,919] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 823033) of binary: /app/anaconda3/envs/python31013llama/bin/python
Traceback (most recent call last):
File "/app/anaconda3/envs/python31013llama/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2024-12-20_01:35:33
host : hbjj
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 823034)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2024-12-20_01:35:33
host : hbjj
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 823035)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-12-20_01:35:33
host : hbjj
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 823033)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Process ForkServerProcess-8:
Process ForkServerProcess-3:
Process ForkServerProcess-4:
Process ForkServerProcess-6:
Process ForkServerProcess-7:
Process ForkServerProcess-2:
Process ForkServerProcess-5:
Process ForkServerProcess-9:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "<string>", line 2, in get
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "<string>", line 2, in get
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "<string>", line 2, in get
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "<string>", line 2, in get
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
Traceback (most recent call last):
File "<string>", line 2, in get
Traceback (most recent call last):
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "<string>", line 2, in get
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
EOFError
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
EOFError
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "<string>", line 2, in get
File "<string>", line 2, in get
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
EOFError
/app/anaconda3/envs/python31013llama/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '