Cluster Node Synchronization API

Function

Synchronizes cluster nodes.

This API does not need to be called by users. It is a communication API between the Coordinator and Controller.

Format

Operation type: POST

URL: https://{ip}:{port}/v1/instances/refresh

Preferably set {ip} to {manage_ip} in Starting Coordinator. If the parameter is not configured, set {ip} to the value of the manage_ip parameter in the ms_coordinator.json configuration file.
Preferably set {port} to {manage_port} in Starting Coordinator. If the parameter is not configured, set {port} to the value of the manage_port parameter in the ms_coordinator.json configuration file.

Request Parameters

Parameter	Type	Description
ids	uint64_t[]	Mandatory. ID list of nodes to be synchronized.
instances	object[]	Mandatory. Detailed information about the nodes to be synchronized.
id	uint64_t	Mandatory. Node ID.
ip	String	Mandatory. Node IP address.
port	String	Mandatory. Value range: ["1024", "65535"] Node data port.
metric_port	String	Mandatory. Value range: ["1024", "65535"] Metrics port of the node.
model_name	String	Mandatory. Name of the model loaded to a node.
static_info	Object	Mandatory. Static node information.
group_id	uint64_t	Mandatory. ID of the group to which a node belongs.
max_seq_len	uint32_t	Mandatory. The value must be greater than 0. Maximum queue length of a node.
max_output_len	uint32_t	Mandatory. The value range is [1, max_seq_len – 1]. Maximum inference output length of a node.
total_slots_num	uint32_t	Mandatory. Value range: [1, 5000]. Maximum number of inference jobs on a node.
total_block_num	uint32_t	Mandatory. The value must be greater than 0. Maximum number of memory blocks on a node.
block_size	uint32_t	Mandatory. Value range: [1, 128]. Size of a memory block.
label	uint32_t	Mandatory. Valid only in prefill-decode disaggregation scenarios. Node label. The values are as follows: 2: prefill node 3: decode node
role	uint32_t	Mandatory. Node label. The values are as follows: 80: prefill node 68: decode node 85: Undef node
dynamic_info	Object	Mandatory. Dynamic node information.
avail_slots_num	uint32_t	Mandatory. Value range: [0, total_slots_num]. Number of remaining available inference jobs on a node.
avail_block_num	uint32_t	Mandatory. Value range: [0, total_block_num]. Number of available memory blocks on a node.
peers	uint64_t[]	IDs of all prefill nodes connected to the decode node. This field is not required for a prefill node. This field is mandatory for a decode node.

Usage Example

Request example:

POST: https://{ip}:{port}/v1/instances/refresh

Request body:

{
    "ids": [
        0,1
    ],
    "instances": [
        {
            "id": 0,
            "ip": "0.0.0.0",
            "port": "1025",
            "metric_port": "1026",
            "model_name": "your_model_name",
            "static_info": {
                "group_id": 0,
                "max_seq_len": 2048,
                "max_output_len": 512,
                "total_slots_num": 200,
                "total_block_num": 1024,
                "block_size": 128,
                "label": 2,
                "role": 80
            },
            "dynamic_info": {
                "avail_slots_num": 200,
                "avail_block_num": 1024
            }
        },
        {
            "id": 1,
            "ip": "0.0.0.0",
            "port": "1025",
            "metric_port": "1026",
            "model_name": "your_model_name",
            "static_info": {
                "group_id": 0,
                "max_seq_len": 2048,
                "max_output_len": 512,
                "total_slots_num": 200,
                "total_block_num": 1024,
                "block_size": 128,
                "label": 3,
                "role": 68
            },
            "dynamic_info": {
                "avail_slots_num": 200,
                "avail_block_num": 1024,
                "peers": [
                    0
                ]
            }
        }
    ]
}

Response example:

In normal cases, there is no response message body.

Output Description

If the JSON format and fields in the request body are correct, the 200 status code is returned. Otherwise, the 400 status code is returned.

Parent topic: Intra-cluster Communication APIs