Port Binding Failure on the Parameter Plane (EJ0003)

Symptom

The CANN logs contain the keyword "Please check the port status and whether the port is being used by other process.", as shown in the following. In addition, port binding may fail in the communicator cluster negotiation phase. You can determine the port binding failure based on the socket type in the error logs. If the type is 0 or 1, the port binding fails for the parameter plane. If the type is 2, the port binding fails for the host NIC during communicator cluster negotiation. For details, see Port Binding Failure on the Server (EJ0003).

[ERROR] HCCL(1009464,all_reduce_test):2000-03-15-00:41:48.470.172 [hccl_socket.cc:110] [1009464][InitGroupStage][RanktableDetect] socket type[0], listen on ip[192.168.2.199] and specific port[16666] fail. Please check the port status and whether the port is being used by other process.

Possible Cause

When setting up a connection on the communicator operator parameter plane, the rank or process needs to bind a port of the NIC on the device. However, the port has been occupied by another process.

Solution

By default, port 16666 needs to be bound when HCCL uses the port of the NIC on the device. Therefore, if multiple processes are executed on the same device and all these processes call the communication operator API of HCCL, the failure occurs because the port has been bound by another process.

In this case, you can check whether running multiple processes on the same device meets the task expectation. If yes, you can configure the HCCL_NPU_SOCKET_PORT_RANGE environment variable to enable the multi-process scenario. An example is as follows:

export HCCL_NPU_SOCKET_PORT_RANGE="auto"