---
title: ModelArts多机多卡训练GPT2-13B大模型时报错EI0002: The wait execution of the Notify register times out-昇腾社区
description: ModelArts上多机多卡训练GPT2-13B大模型时报错EI0002: The wait execution of the Notify register times out. Reason: The Notify register has not received the Notify record from remote rank [1].base information: [stream
keywords: ModelArts多机多卡;训练GPT2-13B
url: https://www.hiascend.com/document/caselibrary/detail/case_9526
section: (其他)
---

# ModelArts多机多卡训练GPT2-13B大模型时报错EI0002: The wait execution of the Notify register times out-昇腾社区

URL: https://www.hiascend.com/document/caselibrary/detail/case_9526
描述: ModelArts上多机多卡训练GPT2-13B大模型时报错EI0002: The wait execution of the Notify register times out. Reason: The Notify register has not received the Notify record from remote rank [1].base information: [stream
关键词: ModelArts多机多卡;训练GPT2-13B

昇腾文档 [了解详情](https://www.hiascend.com/document)

故障案例 [了解详情](https://www.hiascend.com/document/caselibrary)

ModelArts多机多卡训练GPT2-13B大模型时报错EI0002: The wait execution of the Notify register times out

ModelArts多机多卡训练GPT2-13B大模型时报错EI0002: The wait execution of the Notify register times out

2024/02/22

1.4k

问题信息

| 问题来源 | 产品大类 | 产品子类 | 关键字 |
| --- | --- | --- | ---|
| 官方 | 模型训练 | MindSpore | ModelArts多机多卡、训练GPT2-13B |

#### 问题现象描述

ModelArts上多机多卡训练GPT2-13B大模型时报错EI0002: The wait execution of the Notify register times out. Reason: The Notify register has not received the Notify record from remote rank [1].base information: [streamID:[153], taskID[19], taskType[Notify Wait], tag[HcomAllReduce_5871863451775290338_0].] task information: [notify id:[0x00000000000001c8], stage:[ffffffff], remote rank:[1].]

ms版本：1.9.0

开源代码路径：https://gitee.com/mindspore/mindformers/tree/dev [了解详情](https://gitee.com/mindspore/mindformers/tree/dev)

#### 原因分析

每个NPU的训练样本的数量是不一致的。

集群中某些NPU的执行速度太慢，无法在超时间隔内完成通信操作。

在集群中的某些NPU的执行过程中发生了异常。

#### 排查方法

1.确认每个NPU上训练样本数是否一致；

2.打印HCCL_EXEC_TIMEOUT值，确认是否很小；

3.执行npu-smi info，看NPU是否正常；

#### 解决措施

增加了环境变量HCCL_EXEC_TIMEOUT=7200

本页内容

- 问题信息
- 问题现象描述
- 原因分析
- 排查方法
- 解决措施