CANN版本：7.0.0

驱动版本：23.0.0

pytorch：2.1.0

torch_npu:2.1.0

报错信息：

 

  5%|▌         | 53/1060 [01:50<17:06,  1.02s/it]***** Running Evaluation ***** 

  Num examples = 9900 

  Batch size = 100 

{'loss': 45.2135, 'grad_norm': 1.3439862728118896, 'learning_rate': 2.5e-05, 'epoch': 1.0} 

  5%|▌         | 53/1060 [01:54<17:06,  1.02s/iSaving model checkpoint to ./results/conformer_prediction/GTMGC_Qm9/checkpoint-53 

Configuration saved in ./results/conformer_prediction/GTMGC_Qm9/checkpoint-53/config.json 

Model weights saved in ./results/conformer_prediction/GTMGC_Qm9/checkpoint-53/model.safetensors 

{'eval_loss': 4545259.0, 'eval_mae': 4545259.5, 'eval_mse': 4.4353553458161254e+17, 'eval_rmsd': 6686393.0, 'eval_runtime': 4.0039, 'eval_samples_per_econd': 2472.612, 'eval_steps_per_second': 24.726, 'epoch': 1.0} 

 10%|█         | 106/1060 [02:52<16:42,  1.05s/it]***** Running Evaluation ***** 

  Num examples = 9900 

  Batch size = 100 

{'loss': 15700432316.3774, 'grad_norm': 0.01227150484919548, 'learning_rate': 5e-05, 'epoch': 2.0} 

 10%|█         | 106/1060 [02:56<16:42,  1.05s/Saving model checkpoint to ./results/conformer_prediction/GTMGC_Qm9/checkpoint-106 

Configuration saved in ./results/conformer_prediction/GTMGC_Qm9/checkpoint-106/config.json 

Model weights saved in ./results/conformer_prediction/GTMGC_Qm9/checkpoint-106/model.safetensors 

{'eval_loss': 165.8354034423828, 'eval_mae': 165.8354034423828, 'eval_mse': 12031118.0, 'eval_rmsd': 161.54119873046875, 'eval_runtime': 3.949, 'eval_amples_per_second': 2506.934, 'eval_steps_per_second': 25.069, 'epoch': 2.0} 

 10%|█         | 108/1060 [03:04<50:27,  3.18s/it]  Traceback (most recent call last): 

  File "/root/GTMGC/experiments/conformer_prediction/gtmgc_for_conformer_prediction.py", line 100, in main 

    train_result = trainer.train() 

  File "/root/miniconda3/envs/torch/lib/python3.9/site-packages/transformers/trainer.py", line 1780, in train 

    return inner_training_loop( 

  File "/root/miniconda3/envs/torch/lib/python3.9/site-packages/transformers/trainer.py", line 2118, in _inner_training_loop 

    tr_loss_step = self.training_step(model, inputs) 

  File "/root/miniconda3/envs/torch/lib/python3.9/site-packages/transformers/trainer.py", line 3045, in training_step 

    self.accelerator.backward(loss) 

  File "/root/miniconda3/envs/torch/lib/python3.9/site-packages/accelerate/accelerator.py", line 2001, in backward 

    loss.backward(**kwargs) 

  File "/root/miniconda3/envs/torch/lib/python3.9/site-packages/torch/_tensor.py", line 492, in backward 

    torch.autograd.backward( 

  File "/root/miniconda3/envs/torch/lib/python3.9/site-packages/torch/autograd/__init__.py", line 251, in backward 

    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass 

RuntimeError: Function 'CdistBackward0' returned nan values in its 0th output. 

 

During handling of the above exception, another exception occurred: 

 

Traceback (most recent call last): 

  File "/root/miniconda3/envs/torch/lib/python3.9/runpy.py", line 197, in _run_module_as_main 

    return _run_code(code, main_globals, None, 

  File "/root/miniconda3/envs/torch/lib/python3.9/runpy.py", line 87, in _run_code 

    exec(code, run_globals) 

  File "/root/GTMGC/experiments/conformer_prediction/gtmgc_for_conformer_prediction.py", line 116, in <module> 

    main() 

  File "/root/GTMGC/experiments/conformer_prediction/gtmgc_for_conformer_prediction.py", line 105, in main 

    train_result = trainer.train(resume_from_checkpoint=resume_from_checkpoint) 

  File "/root/miniconda3/envs/torch/lib/python3.9/site-packages/transformers/trainer.py", line 1752, in train 

    self._load_from_checkpoint(resume_from_checkpoint) 

  File "/root/miniconda3/envs/torch/lib/python3.9/site-packages/transformers/trainer.py", line 2346, in _load_from_checkpoint 

    raise ValueError(f"Can't find a valid checkpoint at {resume_from_checkpoint}") 

ValueError: Can't find a valid checkpoint at False