[2025-03-05 14:23:16,980] [287] [281465984119136] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281473588140304 id of vcache is 281473588140208 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown 

  warnings.warn('resource_tracker: There appear to be %d ' 

/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown 

  warnings.warn('resource_tracker: There appear to be %d ' 

Daemon is killing... 

类型是bf16，14B的可以正常跑起，32B-w8a8改成fp16也可以跑起。

config.json配置如下：

{ 

    "architectures": [ 

        "Qwen2ForCausalLM" 

    ], 

    "attention_dropout": 0.0, 

    "bos_token_id": 151643, 

    "eos_token_id": 151643, 

    "hidden_act": "silu", 

    "hidden_size": 5120, 

    "initializer_range": 0.02, 

    "intermediate_size": 27648, 

    "max_position_embeddings": 131072, 

    "max_window_layers": 64, 

    "model_type": "qwen2", 

    "num_attention_heads": 40, 

    "num_hidden_layers": 64, 

    "num_key_value_heads": 8, 

    "rms_norm_eps": 1e-05, 

    "rope_theta": 1000000.0, 

    "sliding_window": 131072, 

    "tie_word_embeddings": false, 

    "torch_dtype": "bfloat16", 

    "transformers_version": "4.43.1", 

    "use_cache": true, 

    "use_sliding_window": false, 

    "vocab_size": 152064, 

    "quantize": "w8a8", 

    "quantization_config": { 

        "group_size": 0, 

        "kv_quant_type": null, 

        "fa_quant_type": null, 

        "w_bit": 8, 

        "a_bit": 8, 

        "dev_type": "npu", 

        "fraction": 0.01, 

        "act_method": 1, 

        "co_sparse": false, 

        "anti_method": "", 

        "disable_level": "L0", 

        "do_smooth": false, 

        "use_sigma": false, 

        "sigma_factor": 3.0, 

        "is_lowbit": false, 

        "mm_tensor": false, 

        "w_sym": true, 

        "open_outlier": true, 

        "is_dynamic": false 

    } 

}