/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use-env is set by default in torchrun.
If your script expects `--local-rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions

  warnings.warn(
[2024-08-26 09:19:00,493] torch.distributed.run: [WARNING]
[2024-08-26 09:19:00,493] torch.distributed.run: [WARNING] *****************************************
[2024-08-26 09:19:00,493] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2024-08-26 09:19:00,493] torch.distributed.run: [WARNING] *****************************************
/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:299: ImportWarning:
    *************************************************************************************************************
    The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
    The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
    The backend in torch.distributed.init_process_group set to hccl now..
    The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
    The device parameters have been replaced with npu in the function below:
    torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
    *************************************************************************************************************

  warnings.warn(msg, ImportWarning)
/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/home/data/szsys/ModelLink/megatron/core/transformer/custom_layers/transformer_engine.py:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import packaging
/home/data/szsys/ModelLink/megatron/core/transformer/custom_layers/transformer_engine.py:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import packaging
/home/data/szsys/ModelLink/megatron/core/transformer/custom_layers/transformer_engine.py:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import packaging
/home/data/szsys/ModelLink/megatron/core/transformer/custom_layers/transformer_engine.py:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import packaging
/home/data/szsys/ModelLink/megatron/core/transformer/custom_layers/transformer_engine.py:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import packaging
/home/data/szsys/ModelLink/megatron/core/transformer/custom_layers/transformer_engine.py:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import packaging
/home/data/szsys/ModelLink/megatron/core/transformer/custom_layers/transformer_engine.py:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import packaging
/home/data/szsys/ModelLink/megatron/core/transformer/custom_layers/transformer_engine.py:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import packaging
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/py38_cpu/fusion_attention_v2/build.ninja...
Building extension module fusion_attention_v2...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
Loading extension module fusion_attention_v2...
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/py38_cpu/fusion_attention_v2/build.ninja...
Building extension module fusion_attention_v2...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
Loading extension module fusion_attention_v2...
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/py38_cpu/fusion_attention_v2/build.ninja...
Building extension module fusion_attention_v2...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
Loading extension module fusion_attention_v2...
Loading extension module fusion_attention_v2...
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/py38_cpu/fusion_attention_v2/build.ninja...
Building extension module fusion_attention_v2...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
Loading extension module fusion_attention_v2...
Loading extension module fusion_attention_v2...
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/py38_cpu/fusion_attention_v2/build.ninja...
Building extension module fusion_attention_v2...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
Loading extension module fusion_attention_v2...
Loading extension module fusion_attention_v2...
[2024-08-26 09:19:10,529] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 155074) of binary: /root/miniconda3/envs/szsys_py38/bin/python
Traceback (most recent call last):
  File "/root/miniconda3/envs/szsys_py38/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/szsys_py38/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch/distributed/launch.py", line 196, in <module>
    main()
  File "/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch/distributed/launch.py", line 192, in main
    launch(args)
  File "/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch/distributed/launch.py", line 177, in launch
    run(args)
  File "/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/szsys_py38/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=======================================================
pretrain_gpt.py FAILED
-------------------------------------------------------
Failures:
[1]:
  time      : 2024-08-26_09:19:10
  host      : 17a93e2dcd33
  rank      : 1 (local_rank: 1)
  exitcode  : -6 (pid: 155075)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 155075
[2]:
  time      : 2024-08-26_09:19:10
  host      : 17a93e2dcd33
  rank      : 2 (local_rank: 2)
  exitcode  : -6 (pid: 155076)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 155076
[3]:
  time      : 2024-08-26_09:19:10
  host      : 17a93e2dcd33
  rank      : 3 (local_rank: 3)
  exitcode  : -6 (pid: 155077)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 155077
[4]:
  time      : 2024-08-26_09:19:10
  host      : 17a93e2dcd33
  rank      : 4 (local_rank: 4)
  exitcode  : -6 (pid: 155078)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 155078
[5]:
  time      : 2024-08-26_09:19:10
  host      : 17a93e2dcd33
  rank      : 5 (local_rank: 5)
  exitcode  : -6 (pid: 155079)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 155079
[6]:
  time      : 2024-08-26_09:19:10
  host      : 17a93e2dcd33
  rank      : 6 (local_rank: 6)
  exitcode  : -6 (pid: 155080)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 155080
[7]:
  time      : 2024-08-26_09:19:10
  host      : 17a93e2dcd33
  rank      : 7 (local_rank: 7)
  exitcode  : -6 (pid: 155081)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 155081
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-08-26_09:19:10
  host      : 17a93e2dcd33
  rank      : 0 (local_rank: 0)
  exitcode  : -6 (pid: 155074)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 155074
=======================================================