MindIE多机多卡启动服务时,每台机器显示sucess,主节点和子节点无法互相通信
收藏回复举报
MindIE多机多卡启动服务时,每台机器显示sucess,主节点和子节点无法互相通信
t('forum.solved') 已解决
发表于2025-03-06 15:45:54
0 查看

部署模型:deepseek-R1-W8A8

机器数目:4 (每个910b4)

启动命令与教程相同

主节点报错:

UNKNOWN:No address added out of total 1 resolved for '7.242.111.115:1120' {created_time:"2025-03-06T15:13:29.54073743+08:00", file_line:967, file:"/usr1/MindIE-LLM_BatchScheduler/3rdparty/grpc/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc", children:[FAILED_PRECONDITION:Error in bind: Cannot assign requested address]} 

[mindie-server] [bind port “主节点:1120”] [success]

--------------------------------------------------------------------

[model_backend] Waiting to receive all register msg from all slave nodes. 

[mindie-server] [ERROR] [block_num_waiter.cpp:67] : [MIE04E120507] [model_backend] The received node [子节点ip] status is error 

子节点报错:

[MIE04E120514] [model_backend] RPC failed with status code[4], and error message: Deadline Exceeded 

Daemon start success! 

注:ranktable经查询,IP地址、device_ip 全都无误。4台机器防火墙属于关闭状态。在物理机上使用ping、nc能够进行通信。

通过命令for i in {0..7}; do hccn_tool -i $i -lldp -g | grep Ifname; done  

# 检查链接情况 

for i in {0..7}; do hccn_tool -i $i -link -g ; done 

# 检查网络健康情况 

for i in {0..7}; do hccn_tool -i $i -net_health -g ; done 

# 查看侦测ip的配置是否正确 

for i in {0..7}; do hccn_tool -i $i -netdetect -g ; done 

# 查看网关是否配置正确 

for i in {0..7}; do hccn_tool -i $i -gateway -g ; done

检查无误,但依旧无法进行多机部署!

我要发帖子