Job Information

job-summary-<Job-Name>

Table 1 job-summary-<Job-Name> ConfigMap

Parameter	Description	Value
hccl.json	Chip communication information used by the job. Can be escaped to JSON format, with the following field descriptions: `status`: Whether the job RankTable has been generated. `initializing`: Still allocating devices for the job, RankTable not yet generated. `complete`: Once the RankTable is generated, the status immediately changes to `complete`, and other fields such as `server_list` appear synchronously. `server_list`: Job device allocation status. `device`: Records NPU allocation, NPU IP and rank_id information. `device_id`: Device ID of the NPU. `device_ip`: Device IP of the NPU. `rank_id`: Training Rank ID corresponding to the NPU. `super_device_id`: Unique identifier of the NPU within the SuperPoD. `server_id`: AI Server identifier, globally unique. `server_name`: Node name. `server_sn`: SN number of the node. The device SN must exist. If it does not, contact Huawei technical support. `host_ip`: Host IP. `super_pod_id`: SuperPoD ID. `pod_name`: Pod name. `container_ids`: ID mapping table for all containers in the Pod. `server_count`: Number of nodes used by the job. `version`: Version information. `total`: Number of ConfigMaps.	String
job_id	K8s ID information of the job.	String
operator	`add`: Status updates to `add` after receiving the job adding command. `delete`: Status updates to delete after receiving the job deletion command.	String
deleteTime	Time when the job was deleted.	String
sharedTorIp	Shared switch information used by the job.	String
masterAddr	`MASTER_ADDR` value specified during PyTorch training.	String
total	Number of ConfigMaps.	String
time	Job start time.	String
framework	Framework used by the job.	String
job_status	Job status: `pending` `running` `complete` `failed`	String
job_name	Job name.	String
cm_index	Sequence number of the current ConfigMap.	String
sid	User-defined job ID	String

current-job-statistic

Used to display statistical information of current jobs in the cluster. Detailed information is recorded in the /var/log/mindx-dl/clusterd/event_job.log file. Due to the capacity limit of K8s ConfigMap, the maximum number of cluster jobs supported for statistics is approximately 10,000. When the log file reaches 20 MB, automatic dumping is triggered, and a maximum of 5 dump logs are saved. The maximum retention period for dump logs is 40 days.

Parameter	Description
data	-
- ID	Job ID assigned by the K8s cluster.
- customID	User-defined Job ID. It is not displayed if the content is empty.
- cardNum	Number of cards used by the job. It is not displayed if the content is empty.
- podFirstRunTime	Time when all Pods of the job first entered the running state. It is not displayed if the content is empty.
- stopTime	Time when all Pods of the job were completed or forcibly deleted. It is not displayed if the content is empty.
- podLastRunTime	Time when all Pods of the job last recovered to the running state. It is not displayed if the content is empty.
- podLastFaultTime	Time when some or all Pods of the job last failed. It is not displayed if the content is empty.
- podFaultTimes	Number of Pod rescheduling times caused by job faults. It is not displayed if the count is 0.
totalJob	Total number of jobs in the current cluster.

scheduling-exception-report

This ConfigMap is located in the cluster-system namespace. It is used to display information about jobs with scheduling exceptions in the cluster, helping users quickly locate the causes of job scheduling failures.

Table 7 scheduling-exception-report ConfigMap

Parameter	Description	Value
<jobName>.<jobUID>	Key for job exception information, composed of the job name and job UID.	String
- jobName	Job Name.	String
- jobType	Job type, for example, vcjob, acjob, etc.	String
- nameSpace	Namespace where the job resides.	String
- conditions	Details of job exception conditions.	Object
-- status	Job status. `JobEmptyStatus`: Job status is empty. `JobInitialized`: Job has been initialized. JobFailed: Job failed. `PodGroupCreated`: PodGroup has been created. `PodGroupPending`: PodGroup is in Pending status. `PodGroupInqueue`: PodGroup is in Inqueue status. `PodGroupUnknown:` PodGroup status is unknown. `PodGroupRunning`: PodGroup is in Running status.	String
-- reason	Exception reason, including `JobEnqueueFailed`, `JobValidateFailed`, `NodePredicateFailed`, `BatchOrderFailed`, `NotEnoughResources`, `PodPending`, `PodFailed`, `PgNotInitialized`, `JobNoInitialized`, etc.	String
-- message	Detailed exception information, including fault description and troubleshooting suggestions.	String