As the identity decision-making and status management and control center of the entire Server cluster, the Controller provides the following functions: cluster node status management and control, and PD identity decision-making and delivery. The architecture is shown in .
Figure 1 Controller architecture[object Object][object Object]
Before using Controller, prepare the environment.
- Create a MindIE image by referring to .
- Set the certificate path to the certificate path corresponding to the startup configuration file .
- Complete the setup by referring to .
Startup Configuration File ([object Object])[object Object][object Object]
The following is an example of the [object Object] configuration file. For details about the parameters, see . You can configure the file as needed.
Parameters in the [object Object] configuration file[object Object][object Object]
The fields in the [object Object] configuration file are described as follows. You can configure the fields as needed.
Table 1 Parameter description
[object Object][object Object]
Process Backup File
Service data files can be stored persistently. The path is specified in the [object Object] configuration file. The update period of the process backup file is the same as the period for synchronizing the cluster status. The process backup file contains information such as whether the Server is faulty, Server identity, owner group ID, total number of resources, and whether the Server node has completed identity delivery in the PD disaggregation deployment mode.
- During cluster initialization, if the static configuration collection API or dynamic status collection API of the Server node returns a value other than 200 after the total time for checking the online status of the Server node ends, the Server node fails to go online and is considered as a faulty node.
- During cluster initialization, if the API request for specifying an identity on the Server node fails, the identity fails to be delivered. If the identity is successfully delivered but the Server node fails to be switched over within the total identity initialization check time, the Server node is considered faulty. The possible scenarios are as follows:
- The Server node is specified as a prefill node. After the total identity initialization check time ends, the Server node is still in the
[object Object]or[object Object]state. - The Server node is specified as a D node. After the total identity initialization check time ends, the Server node is still in the
[object Object]state or the D node fails to establish links with all P nodes.
- The Server node is specified as a prefill node. After the total identity initialization check time ends, the Server node is still in the
The following is an example of the process backup file. For details about the parameters, see .
Table 2 Parameters in the process backup file [object Object][object Object]
Cluster Service Status Output File
The file contains the following information:
- Health status of the Coordinator. If the communication API request between the Controller and Coordinator is successful, the Coordinator is in the healthy state.
- Whether the Server node is faulty and the health status of normal nodes.
During cluster initialization, if the static configuration collection API or dynamic status collection API of the Server node returns a value other than 200 after the total time for checking the online status of the Server node ends, the Server node fails to go online and is considered as a faulty node.
During cluster initialization, if the Server node fails to be initialized, the Server node fails to go online and is considered faulty. The possible scenarios are as follows:
- The Server node is specified as a P node. After the total identity initialization check time ends, the Server node is still in the
[object Object]or[object Object]state. - The Server node is specified as a D node. After the total identity initialization check time ends, the Server node is still in the
[object Object]state or the D node fails to establish links with all P nodes.
- The Server node is specified as a P node. After the total identity initialization check time ends, the Server node is still in the
After the cluster is initialized, if the communication API request between the Controller and the normal Server is successful, the Server is considered healthy.
The following is an example of the cluster service status output file. For details about the parameters, see .
Table 3 Parameters in the cluster service status output file [object Object][object Object]
Environment Variables [object Object][object Object]
Currently, the following environment variables are supported.
Table 4 Environment variables supported by the Controller
Model Configuration File
The model configuration file information is as follows:
Table 5 Model configuration file
Hardware Device Information File
The hardware device information file is as follows.
Table 6 Hardware device information file
[object Object]
To ensure high system availability, this feature leverages the etcd distributed lock mechanism to implement Controller failover in the Kubernetes cluster. During initialization, two Controller nodes are started, with their master/standby roles determined through the etcd distributed lock competition. If the master Controller node fails, the standby node automatically takes over, ensuring seamless service switchover. This feature can be used only in MoE EP scenarios.
Constraints
- The master and standby Controller nodes cannot be deployed on the same node.
- This feature takes effect only when the etcd server is correctly deployed. The server requires three replicas to ensure reliability of the etcd cluster.
- The etcd server requires v3.6.
- When PD disaggregation and master and standby Controller nodes are enabled, if master/standby switchover and fault rectification without redundancy (reducing the number of P instances to ensure normal running of D instances due to D instance hardware faults) occur at the same time, fault rectification without redundancy will fail.
Generating an etcd Security Certificate
The master and standby Controller nodes depend on the etcd distributed lock function, which involves communication between different pods in the cluster. You are advised to use the CA certificate for two-way authentication. To configure the certificate, refer to the following procedure.
[object Object]
Prepare the required certificate files in advance and place them under a directory such as
[object Object].[object Object][object Object][object Object][object Object][object Object][object Object][object Object]
[object Object][object Object]
Create the files and directories required by crl.conf.
[object Object][object Object][object Object]Run
[object Object]using the following command to generate files such as the server certificate, client certificate, and CRL.[object Object]If information similar to the following is displayed, the generation is successful:
[object Object]After the command is successfully executed, the following files or directories are generated in the current directory.
[object Object]
Deploying the etcd Server
Load the etcd image.
[object Object][object Object]
Create etcd resources in the cluster.
Create the
[object Object]file.[object Object]Write the following information to the file:
[object Object]Key parameters are as follows:
[object Object]: path to the corresponding node, which must exist.[object Object]: name of the node to be deployed.
Run the following command on the master node of the K8s cluster:
[object Object]If the following information is displayed, the creation is successful:
[object Object]Run the following command to label the three nodes with
[object Object]:[object Object]Run the following command to create the
[object Object]file and configure the etcd certificate on the pod:[object Object]Based on the certificate files generated in , mount the generated file path into the etcd container, and configure etcd to use encrypted communication with
[object Object],[object Object], and[object Object]. Key configuration details are shown in bold below.[object Object]Run the following command on the master node of the K8s cluster:
[object Object]If the following information is displayed, the creation is successful:
[object Object]Run the following command to query pods of the etcd cluster:
[object Object]The command output is similar to the following:
[object Object][object Object]
(Optional) Configuring the RBAC Function for etcd
Role-based access control (RBAC) in etcd is a built-in security mechanism that manages access to key-value data and enables fine-grained control. With RBAC, administrators can assign specific permissions to different users or roles, ensuring that only authorized users can access and perform operations on critical data.
[object Object]
Run the following commands to create the root user and the root user role, and assign the root role to the root user.
[object Object]Run the following commands to create a client user and grant permissions to the user.
[object Object]Run the following command to enable RBAC:
[object Object][object Object]
Configuring and Starting MindIE
Configure certificates on the Controller.
Based on related certificate files generated in , mount the file generation path to the Controller container, that is, modify the
[object Object]file.[object Object]Configure the
[object Object]file.Change the value of
[object Object]to[object Object]to make the CA authentication process take effect.Configure the certificates related to
[object Object]and[object Object].Change the value of
[object Object]to[object Object]and configure the generated client certificate in[object Object].[object Object]
Set anti-affinity scheduling for the Controller.
Anti-affinity scheduling is configured by using the
[object Object]file. By default, anti-affinity scheduling is not configured. In this case, two Controller resources are scheduled to the same node. To configure anti-affinity scheduling, paste the following content under the[object Object]field in the file so that the master and standby Controller nodes can be scheduled to different nodes.[object Object][object Object]
Run the following command to set the same label for the two Controller nodes to be deployed:
[object Object][object Object]: resource type.[object Object]: node name. Set it as required.[object Object]: label key and value. Set them as required.[object Object]
Set the accelerator parameter under the
[object Object]field in the[object Object]file to the preceding node label.
In the
[object Object]configuration file, configure the following parameters to allow two Controller nodes to be deployed:[object Object][object Object]: disabled[object Object]: enabled[object Object]
MindIE Motor provides the instance-level job reconfiguration function after a node is faulty. No matter whether the hardware is faulty or the software exits, instance-level rescheduling is supported and instances can be automatically restored within 10 minutes. This feature can be used only in MoE EP scenarios.
Constraints
The MindCluster service needs to be deployed and the job rescheduling capability (enabled by default in the deployment script) needs to be started. The reconfiguration is classified into two types: with redundant nodes and without redundant nodes.
- With redundancy: If the redundant nodes are sufficient to restore the faulty instances, only the faulty prefill and decode instances are restored.
- Without redundancy: If a node is faulty, the instance cannot be directly restored. In this case, you need to release other instances based on the scaling template to restore the faulty instance. The current restoration policy is reducing the number of prefill instances to ensure the normal running of decode instances. That is, if a decode instance is faulty and the number of prefill instances is greater than 1, release certain prefill instances to obtain redundant nodes to restore the decode instance.
Procedure
Enable the job rescheduling function.
This function is configured in the
[object Object]file. The key configuration parameters are as follows in bold:[object Object]: The value is[object Object], indicating that the job rescheduling function is enabled.[object Object]: indicates the number of rescheduling times. The value range is [7200, 10000].[object Object]: The value is[object Object].
[object Object]Open the
[object Object]configuration file and set the port number for connecting to the MindCluster. The default value is[object Object].[object Object]Go to the
[object Object]directory and run the following command to start the service. When a fault occurs, the function of reconfiguration upon a fault is automatically triggered.[object Object]
Function
Queries the service startup status.
API Format
Operation type: GET
[object Object]
Request Parameters
None
Usage Example
Request:
Response:
Output Description
[object Object]indicates that the service has started successfully.- If no response is returned, the service is not started.
Function:
Checks whether the service status is normal.
[object Object]
API Format
Operation type: GET
[object Object]
Request Parameters
None
Usage Example
Request:
Response:
Output Description
[object Object]indicates that the service status is normal.- If no response is returned, the service is abnormal.
After the Controller is started, the following error information may be displayed during running.
Table 1 Error information