【问题现象】
模型训练时，watchdog超时报错
【版本】
torch 2.1.0
torch-npu 2.1.0.post8
MindSpeed-MM
【报错日志信息】
INFO:mindspeed.mn.data.data_utils.func_utils.template:Replace eos token: <|im_end|>
2025-05-12 16:55:53,972 - template - 275 - INFO - Replace eos token: <|im_end|>
INFO:mindspeed.mn.data.data_utils.func_utils.template:Replace eos token: <|im_end|>
Rank 0, running tokenizer on train_dataset (num_proc=16): 30% | 48000/157712 [07:55<02:22, 772.28 examples/s]
[E compiler_depend.ts:421] [Rank 5] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600196 milliseconds before timing out.
[E compiler_depend.ts:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E compiler_depend.ts:481] To avoid data inconsistency, we are taking the entire process down.
[E compiler_depend.ts:805] [Rank 5] HCCL watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600196 milliseconds before timing out.
[E compiler_depend.ts:421] [Rank 7] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600236 milliseconds before timing out.
[E compiler_depend.ts:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E compiler_depend.ts:481] To avoid data inconsistency, we are taking the entire process down.
[E compiler_depend.ts:805] [Rank 7] HCCL watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600236 milliseconds before timing out.
[E compiler_depend.ts:421] [Rank 4] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600351 milliseconds before timing out.
[E compiler_depend.ts:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E compiler_depend.ts:481] To avoid data inconsistency, we are taking the entire process down.
[E compiler_depend.ts:805] [Rank 4] HCCL watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600351 milliseconds before timing out.
[E compiler_depend.ts:421] [Rank 1] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600495 milliseconds before timing out.
[E compiler_depend.ts:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E compiler_depend.ts:481] To avoid data inconsistency, we are taking the entire process down.
[E compiler_depend.ts:805] [Rank 1] HCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600495 milliseconds before timing out.
点赞
收藏
回复
分享
举报
已解决
【问题现象】
模型训练时，watchdog超时报错
【版本】
torch 2.1.0
torch-npu 2.1.0.post8
MindSpeed-MM
【报错日志信息】
INFO:mindspeed.mn.data.data_utils.func_utils.template:Replace eos token: <|im_end|>
2025-05-12 16:55:53,972 - template - 275 - INFO - Replace eos token: <|im_end|>
INFO:mindspeed.mn.data.data_utils.func_utils.template:Replace eos token: <|im_end|>
Rank 0, running tokenizer on train_dataset (num_proc=16): 30% | 48000/157712 [07:55<02:22, 772.28 examples/s]
[E compiler_depend.ts:421] [Rank 5] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600196 milliseconds before timing out.
[E compiler_depend.ts:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E compiler_depend.ts:481] To avoid data inconsistency, we are taking the entire process down.
[E compiler_depend.ts:805] [Rank 5] HCCL watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600196 milliseconds before timing out.
[E compiler_depend.ts:421] [Rank 7] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600236 milliseconds before timing out.
[E compiler_depend.ts:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E compiler_depend.ts:481] To avoid data inconsistency, we are taking the entire process down.
[E compiler_depend.ts:805] [Rank 7] HCCL watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600236 milliseconds before timing out.
[E compiler_depend.ts:421] [Rank 4] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600351 milliseconds before timing out.
[E compiler_depend.ts:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E compiler_depend.ts:481] To avoid data inconsistency, we are taking the entire process down.
[E compiler_depend.ts:805] [Rank 4] HCCL watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600351 milliseconds before timing out.
[E compiler_depend.ts:421] [Rank 1] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600495 milliseconds before timing out.
[E compiler_depend.ts:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E compiler_depend.ts:481] To avoid data inconsistency, we are taking the entire process down.
[E compiler_depend.ts:805] [Rank 1] HCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600495 milliseconds before timing out.