一台服务器，搭配了 4张 300I pro推理卡

按照指引进行deepseek-32B模型量化到W8A8，msmodelslim/example/DeepSeek/README.md · Ascend/msit - 码云 - 开源中国

运行量化时候产生报错：

python3 quant_deepseek_w8a8.py --model_path /data/deepseek/deepseek-32B --save_path /data/deepseek/deepseek-32B-w8a8/ 

2025-02-17 13:33:08,932 - msmodelslim-logger - WARNING - The current CANN version does not support importing the migration and migration_vit packages. 

Total Process:   0%|                                                                                                                                                                    | 0/4 [00:00<?, ?it/s]/usr/local/lib/python3.11/site-packages/accelerate/utils/modeling.py:1462: UserWarning: Current model requires 1073743872 bytes of buffer for offloaded layers, which seems does not fit any GPU's remaining memory. If you are experiencing a OOM later, please consider using offload_buffers=True. 

  warnings.warn( 

Loading checkpoint shards:  25%|████████████████████████████████████                                                                                                            | 2/8 [00:10<00:32,  5.36s/it] 

Traceback (most recent call last):██████████████████████████████████                                                                                                            | 2/8 [00:10<00:27,  4.53s/it] 

  File "/usr/local/Ascend/msit/msmodelslim/example/DeepSeek/quant_deepseek_w8a8.py", line 39, in <module> 

    model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path=model_path, 

            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained 

    return model_class.from_pretrained( 

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3941, in from_pretrained 

    ) = cls._load_pretrained_model( 

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4415, in _load_pretrained_model 

    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( 

                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 936, in _load_state_dict_into_meta_model 

    set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs) 

  File "/usr/local/lib/python3.11/site-packages/accelerate/utils/modeling.py", line 416, in set_module_tensor_to_device 

    new_value = value.to(device) 

                ^^^^^^^^^^^^^^^^ 

RuntimeError: NPU out of memory. Tried to allocate 272.00 MiB (NPU 0; 21.02 GiB total capacity; 18.82 GiB already allocated; 18.82 GiB current active; 685.46 MiB free; 18.97 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. 

[ERROR] 2025-02-17-13:33:24 (PID:126107, Device:0, RankID:-1) ERR99999 UNKNOWN application exception 

看起来是OOM了，但是不知道如何解决