Release Notes

Version Mapping Description

Product Version Information

Name	MindCluster
Version Number	26.0.0
Version Type	Release Version

NOTE

MindCluster 26.0 version planning: MindCluster 26.0.0, MindCluster 26.1.0, MindCluster 26.2.0, and MindCluster 26.3.0.

Product Version Mapping

Product Name	Version
Ascend HDK	Atlas 350 PCIe card: 1.0.RC1 Other products: 26.0.RC1
CANN	9.0.0

Virus Scan Results

Virus scan passed.

Version Compatibility Description

MindCluster components must be used together. Do not mix components from different versions.

Table 1 Software version compatibility description

MindCluster Software Version	MindCluster Version to Upgrade	CANN Version	Ascend HDK Version	FrameworkPTAdapter Version	MindSpore Version
MindCluster 26.0.0	MindCluster 7.0.RC1 and patch version MindCluster 7.1.RC1 and patch version MindCluster 7.2.RC1 and patch version MindCluster 7.3.0 and patch version	CANN 8.5.0 and patch version CANN 9.0.0 and patch version	Ascend HDK 25.5.0 and patch version Ascend HDK 26.0.RC1 and patch version Ascend HDK 1.0.RC1 and patch version	FrameworkPTAdapter 7.3.0 and patch version FrameworkPTAdapter 26.0.0 and patch version	MindSpore 2.7.2 and patch version MindSpore 2.9.0 and patch version

Version Usage Notes

None

26.0.0 Release Notes

New Features

Feature	Description
MindIO ACP	Supports Async CheckPoint Persistence (ACP) and Training Fault Tolerance (TFT).
MindIO TFT	Supports ACP and TFT. Supports online recovery at specified checkpoint steps after precision anomalies. Supports optimizer-differentiated replica scenarios for the iFLYTEK HULK framework.
MindCluster Ascend FaultDiag	Supports troubleshooting for the Atlas 350 PCIe card. Adds Ascend-faultdiag-toolkit tool to support NPU disconnection and infrastructure link diagnosis. No longer supports building the fault mode library into a binary; the fault mode library is now directly open-sourced.
MindCluster basic components	The service plane network newly supports IPv6. Supports configuring self-healing fault levels or specific fault codes based on task dimensions. Support soft partitioning-based scheduling for Atlas A2/A3 series. Support hard partitioning-based scheduling for Atlas A2/A3 series. MoE EP tasks based on MindIE support switch affinity scheduling. The gRPC heartbeat detection interval of ClusterD is adjusted from the default 5 minutes to 5 seconds. Support post-processing of self-healing faults and rescheduling of non-self-healing faults. Identify whether a fault is a hardware fault based on the cluster dimension; recurring hardware faults are automatically and forcibly isolated to prevent repeated task interruptions. Both automatic forced isolation at the cluster dimension and at the node dimension support configuring an automatic release time. Add network affinity scheduling algorithm of any level, adapted to the Atlas 9000 A3 SuperPoD cluster computing system. New task information subscription interface `SubscribeJobSummarySignalList` is added, supporting returning historical task information on the first subscription. A `ConfigMap` is added to display the reason for task scheduling failures, facilitating fault location. NPU Exporter supports listening and reporting custom metrics through configuration files. NPU Exporter has been refactored to obtain all NPU utilization metrics through a single interface instead of multiple interfaces, thereby preventing data inconsistency. The ClusterD fault notification service supports registration via domain name. The ClusterD job information subscription interface adds a unique job identifier field. NPU Exporter supports metric reporting for the Atlas 350 PCIe card. Ascend Docker Runtime supports the Atlas 350 PCIe card. The Atlas 350 PCIe card supports affinity scheduling, device discovery, RankTable generation, and fault rescheduling. Infer Operator can manage inference tasks through custom CRD.

Key Feature Changes

MindCluster basic components:

The gRPC heartbeat detection interval of ClusterD is adjusted from the default 5 minutes to 5 seconds.
The automatic forced isolation and release of repeatedly faulty chips at the cluster dimension is supported.
For the Atlas 350 PCIe card:
- The task resource request huawei.com/Ascend910 is changed to huawei.com/npu.
- The underlying DCMI is changed to the DCMI V2.

Service Interface Changes

Feature	Interface Change
MindIO ACP	None
MindIO TFT	New `tft_register_exception_handler` interface that registers an exception handler.
MindCluster Ascend FaultDiag	New interfaces related to Ascend-faultdiag-toolkit. For details, see API Description.
MindCluster basic components	Configuration fields for self-healing fault level, fault code, and self-healing duration are added to the task creation interface. Configuration fields for soft partitioning mode, AICore percentage, and high-bandwidth memory size are added to the task creation interface. ClusterD supports configuring the startup switch, triggering frequency, and isolation duration for automatic forced fault isolation. Ascend Device Plugin adds an isolation duration configuration field for automatic forced isolation. Support multi-level network topology configuration and multi-level network affinity configuration for tasks. New task information subscription interface `SubscribeJobSummarySignalList`. New interface for querying the cause of task scheduling exceptions. New file-based custom metric interface. The calculation method of the NPU utilization interface is optimized for NPU Exporter. Basic device information, fault code, and chip name for the Atlas 350 PCIe card are added.

Resolved Issues

When resources such as MindIO processor are released and the program crashes, TaskD Agent cannot exit. A fallback exit mechanism is added.
After training ends, when TaskD Worker calls the mspti_activity_flush_all method, a double free error is reported.
Concurrent map read/write by TaskD Manager causes process crash.
Delayed update of pg cache in ClusterD.
Forced requirement for healthy RoCE network between MindIE instances causes MindIE task scheduling failure.
Pod does not exit after training completion in the user-defined torch log file scenario.
When NodeD is installed and the cluster has more than 1024 cards, the gRPC connection limit of ClusterD is exceeded, preventing other connections from being established

Known Issues

None

Upgrade Impact

Impact of the Upgrade Process on the Current System

None

Impact on the Current System After Upgrade

None

26.0.0 Documents

Document	Content Description	Release Notes
MindCluster Cluster Scheduling User Guide	Introduces cluster scheduling components, feature principles, and usage references, including installation and deployment, integration and adaptation examples, and API references for each component, as well as working principles for some scheduling solutions.	Adds soft partitioning-based scheduling and multi-level scheduling.
MindCluster Fault Diagnosis User Guide	Provides usage guidance for features such as log collection, cleaning and dumping, and troubleshooting.	Adds the Atlas 350 PCIe card fault mode and ascend-faultdiag-toolkit.

List of Fixed Vulnerabilities

None