When training starts, Ascend Operator generates the RankTable file required for collective communication for training jobs. The collective communication domains are built based on the device IDs and IPs in the RankTable file to complete information exchange for collective communication.
- When using Ascend Operator ConfigMap to mount the RankTable, you need to create a ConfigMap named
[object Object]in the YAML when creating a training job, and mount the ConfigMap to the[object Object]path in the training container. Ascend Operator constructs the collective communication RankTable file for the job based on the[object Object]information written by Ascend Device Plugin in the job Pod, writes its content into the ConfigMap, and maps it as the[object Object]file in the training container. - When using shared storage to mount the RankTable, you need to mount a shared storage or local storage directory in the YAML when creating a training job, and mount the directory to the
[object Object]path in the training container. Ascend Operator constructs the collective communication RankTable file for the job based on the[object Object]information written by Ascend Device Plugin or volcano-scheduler in the job Pod, writes its content into the[object Object]file, and maps it as the[object Object]file in the training container. - Different products have different
[object Object]file contents, as detailed below.
The [object Object] file example is as follows:
[object Object]
The following is an example of the [object Object] file:
[object Object]
The [object Object] file example is as follows:
[object Object]