BIOS Configuration

Modifying the CPU Power Consumption Mode

Principle: Generally, the Ascend AI Processor provides two power policy configurations: Efficiency and Performance.

• Efficiency indicates the power saving mode. The CPU supports dynamic frequency and voltage scaling and can dynamically adjust the working frequency based on the load.
• Performance indicates the performance mode. The CPU does not support dynamic frequency scaling and runs at the maximum frequency.

Optimization configuration: Power Policy is set to Performance for better performance.

Disadvantage: Enabling the high-performance mode will trigger high power consumption.

Recommended scenario: In inference scenarios, you are advised to enable the performance mode to improve CPU performance and reduce CPU idle cycles, which improves model performance.

To study how the high-performance function impacts inference performance, the lab utilizes an Atlas 800T A2 server running openEuler 22.03 SP4. Within this environment, the Llama-7B and Qwen2-7B models are deployed to provide a comparative analysis of execution performance before and after the high-performance function is enabled. The following table shows the test data based on MindIE. Because software and hardware configurations vary, the following test data is intended for reference only and should not be considered a performance standard.

Configuration method: Access the BIOS through the BMC and set Power Policy to Performance under Advanced > Performance Config > Power Policy.

Modifying the Memory Refresh Rate

Principle: The DRAM uses a capacitor to store data. Due to electric leakage of the capacitor, charges are discharged after a period of time. As a result, the data cannot be stored for a long time. Therefore, continuous charging is required. This is called a refresh operation. The refresh operation and read/write operation cannot be performed at the same time. That means that the refresh operation affects the memory performance. The BIOS includes an Auto setting for the memory refresh rate. This feature automatically adjusts the rate based on the current operating temperature, offering superior memory performance compared to the default 32 ms configuration.

Optimization configuration: The memory refresh rate is set to Auto.

Recommended scenario: The memory refresh rate is dynamically adjusted, which can improve memory copy performance and model performance.

To study how different memory refresh rates impact the inference performance, the lab utilizes an Atlas 800T A2 server running openEuler 22.03 SP4. Within this environment, the Llama-7B and Qwen2-7B models are deployed to provide a comparative analysis of execution performance. The following table shows the test data based on MindIE. Because software and hardware configurations vary, the following test data is intended for reference only.

Configuration method: Access the BIOS through the BMC and set Custom Refresh Rate to Auto under Advanced > Memory Config > Custom Refresh Rate.

Modifying the CPU Prefetching Configuration

Principle: When reading data from the memory to the high-speed cache of the CPU, the CPU not only reads the data to be accessed this time, but also prefetches the surrounding data of the current data item to the cache according to the locality principle. If the prefetched data is the data to be obtained next time, the performance is improved.

Optimization configuration: CPU prefetching is enabled.

Disadvantage: In scenarios where data is centralized and the prefetch hit ratio remains high, you are advised to enable CPU prefetching. Otherwise, you need to disable CPU prefetching.

Recommended scenario: In inference scenarios, you are advised to enable CPU prefetching to improve the CPU data read performance and model performance.

To study how the CPU prefetching function impacts inference performance, the lab utilizes an Atlas 800T A2 server running openEuler 22.03 SP4. Within this environment, the Llama-7B and Qwen2-7B models are deployed to provide a comparative analysis of execution performance before and after the CPU prefetching function is enabled. The following table shows the test data based on MindIE. Because software and hardware configurations vary, the following test data is intended for reference only.

Configuration method: Access the BIOS through the BMC and set CPU Prefetching Configuration to Enabled under Advanced > MISC Config > CPU Prefetching Configuration.

Disabling SMMU

Principle: The SMMU implements the conversion from virtual addresses to physical addresses. However, the SMMU may increase additional overhead and latency, which deteriorates the system performance.

Optimization configuration: SMMU is disabled.

Recommended scenario: SMMU is disabled in non-VM scenarios (bare metal and Docker) to improve model performance.

To study how the SMMU impacts inference performance, the lab utilizes an Atlas 800T A2 server running openEuler 22.03 SP4. Within this environment, the Llama-7B and Qwen2-7B models are deployed to provide a comparative analysis of execution performance before and after the SMMU is enabled. The following table shows the test data based on MindIE. Because software and hardware configurations vary, the following test data is intended for reference only.

Configuration method: Access the BIOS through the BMC and set Support Smmu to Disabled under Advanced > MISC Config > Support Smmu.