E0309 21: 11: 36.962072875 12406 chttp2_server.cc: 1051] UNKNOWN: No address added out of total 1 resolved
for '172.16.190.11:1120' {
created_time: "2025-03-09T21:11:36.961808011+08:00",
file_line: 967,
file: "/usr1/MindIE-LLM_BatchScheduler/3rdparty/grpc/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc",
children: [FAILED_PRECONDITION: Error in bind: Cannot assign requested address]
}
172.16.190.11是主节点
172.16.190.12是子节点
双机部署,172.16.190.12上的配置文件怎么填写?
子节点目前运行后,报错:
[2025-03-09 21:21:49,158] [14828] [281462240964960] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-03-09 21:21:49,545] [14823] [281464709050720] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-03-09 21:21:50,326] [14811] [281461001154912] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=223, need_nz=False, matmul_nd_nz=False)
[2025-03-09 21:21:51,331] [14817] [281457228509536] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-03-09 21:21:52,271] [14821] [281459577385312] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...
Killed
for '172.16.190.11:1120' {
created_time: "2025-03-09T21:11:36.961808011+08:00",
file_line: 967,
file: "/usr1/MindIE-LLM_BatchScheduler/3rdparty/grpc/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc",
children: [FAILED_PRECONDITION: Error in bind: Cannot assign requested address]
}
172.16.190.11是主节点
172.16.190.12是子节点
双机部署,172.16.190.12上的配置文件怎么填写?
子节点目前运行后,报错:
[2025-03-09 21:21:49,158] [14828] [281462240964960] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-03-09 21:21:49,545] [14823] [281464709050720] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-03-09 21:21:50,326] [14811] [281461001154912] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=223, need_nz=False, matmul_nd_nz=False)
[2025-03-09 21:21:51,331] [14817] [281457228509536] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-03-09 21:21:52,271] [14821] [281459577385312] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...
Killed