Port Binding Failure on the Parameter Plane (EI0019)

Symptom

The CANN logs contain the keyword "Please check the port status and whether the port is being used by other process," as shown below. Port binding may also fail in the communicator cluster negotiation phase. You can determine the failure based on the socket type in the error logs. Type 0 or 1 indicates port binding failure on the parameter plane. Type 2 indicates port binding failure on the host NIC during communicator cluster negotiation. For details, see Port Binding Failure on the Server (EI0019).

[ERROR] HCCL(1009464,all_reduce_test):2025-03-15-00:41:48.470.172 [hccl_socket.cc:110] [1009464][InitGroupStage][RanktableDetect] socket type[0], listen on ip[192.168.2.199] and specific port[16666] fail. Please check the port status and whether the port is being used by other process.

Possible Cause

When setting up a connection on the communicator operator parameter plane, the rank or process needs to bind a port of the NIC on the device. However, the port has been occupied by another process.

Solution

By default, port 16666 needs to be bound when HCCL uses the port of the NIC on the device. This failure occurs when multiple processes are executed on the same device and all these processes call the communication operator API of HCCL.

You can check whether running multiple processes on the same device meets the task expectation. If so, you can configure the HCCL_NPU_SOCKET_PORT_RANGE environment variable to enable the multi-process scenario. An example is as follows:

export HCCL_NPU_SOCKET_PORT_RANGE="auto"