[INFO|trainer.py:1779] 2024-05-05 02:55:04,071 >> ***** Running training ***** 

[INFO|trainer.py:1780] 2024-05-05 02:55:04,071 >>   Num examples = 114,599 

[INFO|trainer.py:1781] 2024-05-05 02:55:04,071 >>   Num Epochs = 1 

[INFO|trainer.py:1782] 2024-05-05 02:55:04,071 >>   Instantaneous batch size per device = 1 

[INFO|trainer.py:1783] 2024-05-05 02:55:04,072 >>   Total train batch size (w. parallel, distributed & accumulation) = 1 

[INFO|trainer.py:1784] 2024-05-05 02:55:04,072 >>   Gradient Accumulation steps = 1 

[INFO|trainer.py:1785] 2024-05-05 02:55:04,072 >>   Total optimization steps = 3,000 

[INFO|trainer.py:1786] 2024-05-05 02:55:04,073 >>   Number of trainable parameters = 1,835,008 

  0%|                                                                                                                                                                                   | 0/3000 [00:00<?, ?it/s]Warning: Device do not support double dtype now, dtype cast repalce with float. 

Warning: torch.npu_confusion_transpose is deprecated and will be removed in future version. Use torch_npu.npu_confusion_transpose instead. 

Traceback (most recent call last): 

  File "main_without_tokenizer.py", line 308, in <module> 

    main() 

  File "main_without_tokenizer.py", line 245, in main 

    train_result = trainer.train(resume_from_checkpoint=checkpoint) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/transformers/trainer.py", line 1668, in train 

    ignore_keys_for_eval=ignore_keys_for_eval, 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/transformers/trainer.py", line 1940, in _inner_training_loop 

    tr_loss_step = self.training_step(model, inputs) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/transformers/trainer.py", line 2735, in training_step 

    loss = self.compute_loss(model, inputs) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/transformers/trainer.py", line 2767, in compute_loss 

    outputs = model(**inputs) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl 

    return forward_call(*input, **kwargs) 

  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 968, in forward 

    return_dict=return_dict, 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl 

    return forward_call(*input, **kwargs) 

  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 858, in forward 

    kv_caches=past_key_values, use_cache=use_cache, output_hidden_states=output_hidden_states 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl 

    return forward_call(*input, **kwargs) 

  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 654, in forward 

    use_cache 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/torch_npu/utils/checkpoint.py", line 275, in checkpoint 

    return CheckpointFunction.apply(function, preserve, *args) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/torch_npu/utils/checkpoint.py", line 120, in forward 

    outputs = run_function(*args) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl 

    return forward_call(*input, **kwargs) 

  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 566, in forward 

    use_cache=use_cache 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl 

    return forward_call(*input, **kwargs) 

  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 458, in forward 

    context_layer = self.core_attention(query_layer, key_layer, value_layer, attention_mask) 

  File "/usr/local/python3.7.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl 

    return forward_call(*input, **kwargs) 

  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 318, in forward 

    context_layer = torch.bmm(attention_probs, value_layer.transpose(0, 1)) 

RuntimeError: NPU out of memory. Tried to allocate 4.06 GiB (NPU 2; 32.00 GiB total capacity; 24.68 GiB already allocated; 2.48 GiB free; 28.93 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. 

  0%|                                                                     

【说明】

在微调过程中出现如上报错，求解答是什么原因导致的。npu显存32G。脚本参数设置入下：