(myenv) root@orangepiaipro-20t:/usr/local/Ascend/MindIE-LLM# python   -m examples.run_fa          --model_path /root/DeepSeek-R1-Distill-Llama-8B-OrangePi/deepseek-llama-8B-w8a8/          -
-input_text 'What is deep learning?' --max_output_length 20
path /configs/mki_cfg.ini does not exist
File size is invalid
ParseIniFileToMapNpLog failed!
ParseLogData failed!
[2025-03-01 07:55:35,093] [135882] [255085626875936] [llm] [INFO] [model_runner.py-95] : Skip binding cpu.
[2025-03-01 07:55:36,494] [135882] [255085626875936] [llm] [INFO] [model_runner.py-125] : model_runner.quantize: w8a8, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.float16
[2025-03-01 07:56:01,341] [135882] [255085626875936] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-03-01 07:56:01,348] [135882] [255085626875936] [llm] [INFO] [model_runner.py-147] : init tokenizer done
CausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[2025-03-01 07:56:01,419] [135882] [255085626875936] [llm] [INFO] [causal_lm.py-62] : NPUSocInfo(soc_name='', soc_version=240, need_nz=False, matmul_nd_nz=False)
Traceback (most recent call last):
  File "/usr/local/miniconda3/envs/myenv/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/miniconda3/envs/myenv/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/Ascend/MindIE-LLM/examples/run_fa.py", line 340, in <module>
    fa_runner = FARunner(**input_dict)
  File "/usr/local/Ascend/MindIE-LLM/examples/run_fa.py", line 111, in __init__
    self.model.load_weights()
  File "/usr/local/Ascend/MindIE-LLM/atb_llm/runner/model_runner.py", line 183, in load_weights
    self.attn_mask = self.model.attn_mask
  File "/usr/local/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'LlamaForCausalLM' object has no attribute 'attn_mask'
[ERROR] 2025-03-01-07:57:06 (PID:135882, Device:0, RankID:-1) ERR99999 UNKNOWN application exception
(myenv) root@orangepiaipro-20t:/usr/local/Ascend/MindIE-LLM# python   -m examples.run_fa_edge          --model_path /root/DeepSeek-R1-Distill-Llama-8B-OrangePi/deepseek-llama-8B-w8a8/
    --input_text 'What is deep learning?' --max_output_length 20
path /configs/mki_cfg.ini does not exist
File size is invalid
ParseIniFileToMapNpLog failed!
ParseLogData failed!
[2025-03-01 08:03:02,180] [136412] [255085730172960] [llm] [INFO] [model_runner.py-95] : Skip binding cpu.
[2025-03-01 08:03:03,694] [136412] [255085730172960] [llm] [INFO] [model_runner.py-125] : model_runner.quantize: w8a8, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.float16
[2025-03-01 08:03:31,369] [136412] [255085730172960] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-03-01 08:03:31,386] [136412] [255085730172960] [llm] [INFO] [model_runner.py-147] : init tokenizer done
CausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
[2025-03-01 08:03:31,455] [136412] [255085730172960] [llm] [INFO] [causal_lm.py-62] : NPUSocInfo(soc_name='', soc_version=240, need_nz=False, matmul_nd_nz=False)
[2025-03-01 08:04:53,477] [136412] [255085730172960] [llm] [INFO] [model_runner.py-218] : model:
 LlamaForCausalLM(
  (rotary_embedding): PositionRotaryEmbedding()
  (model): FlashLlamaModel(
    (embed_tokens): TensorParallelEmbedding()
    (layers): ModuleList(
      (0-31): 32 x FlashLlamaLayer(
        (self_attn): FlashLlamaAttention(
          (query_key_value): TensorParallelColumnLinear(
            (linear): W8A8LinearStatic()
          )
          (o_proj): TensorParallelRowLinear(
            (linear): W8A8LinearStatic()
          )
        )
        (mlp): LlamaMLP(
          (gate_up_proj): TensorParallelColumnLinear(
            (linear): W8A8LinearStatic()
          )
          (down_proj): TensorParallelRowLinear(
            (linear): W8A8LinearStatic()
          )
        )
        (input_layernorm): RMSNormWrapper(
          (ori): RMSNorm()
          (anti): RMSNormBias()
        )
        (post_attention_layernorm): RMSNormWrapper(
          (ori): RMSNorm()
          (anti): RMSNormBias()
        )
      )
    )
    (norm): RMSNorm()
  )
  (lm_head): TensorParallelHead(
    (linear): FastLinear()
  )
  (attn_mask): AttentionMask()
)
[2025-03-01 08:04:53,477] [136412] [255085730172960] [llm] [INFO] [run_fa_edge.py-133] : ---------------begin warm_up---------------
/usr/local/miniconda3/envs/myenv/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:601: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/usr/local/miniconda3/envs/myenv/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:606: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.95` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(
/usr/local/miniconda3/envs/myenv/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:623: UserWarning: `do_sample` is set to `False`. However, `top_k` is set to `1000` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_k`.
  warnings.warn(
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:None for open-end generation.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
[2025-03-01 08:05:04,238] [136412] [255085730172960] [llm] [INFO] [run_fa_edge.py-140] : ---------------end warm_up---------------
[2025-03-01 08:05:04,238] [136412] [255085730172960] [llm] [INFO] [run_fa_edge.py-143] : ---------------begin inference---------------
Setting `pad_token_id` to `eos_token_id`:None for open-end generation.
/usr/local/miniconda3/envs/myenv/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:601: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/usr/local/miniconda3/envs/myenv/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:606: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.95` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(
/usr/local/miniconda3/envs/myenv/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:623: UserWarning: `do_sample` is set to `False`. However, `top_k` is set to `1000` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_k`.
  warnings.warn(
Setting `pad_token_id` to `eos_token_id`:None for open-end generation.
[2025-03-01 08:05:16,300] [136412] [255085730172960] [llm] [INFO] [run_fa_edge.py-186] : inputs.input_ids.shape=torch.Size([1, 6])
[2025-03-01 08:05:16,300] [136412] [255085730172960] [llm] [INFO] [run_fa_edge.py-190] : Question: What is deep learning?
[2025-03-01 08:05:16,300] [136412] [255085730172960] [llm] [INFO] [run_fa_edge.py-191] : Answer:  coloc habit Virsame自治 Sanatitersselenantabrwartадуerezаду ais室STEMSTEMerezSTEM
[2025-03-01 08:05:16,301] [136412] [255085730172960] [llm] [INFO] [run_fa_edge.py-192] : Input token num: 6
[2025-03-01 08:05:16,301] [136412] [255085730172960] [llm] [INFO] [run_fa_edge.py-193] : Generate token num: 20
[2025-03-01 08:05:16,301] [136412] [255085730172960] [llm] [INFO] [run_fa_edge.py-195] : ---------------end inference---------------
[2025-03-01 08:05:16,301] [136412] [255085730172960] [llm] [INFO] [run_fa_edge.py-203] : Decode average time(ms): 582.8551744159898
[2025-03-01 08:05:16,301] [136412] [255085730172960] [llm] [INFO] [run_fa_edge.py-207] : Total time(ms): 12013.3638381958, First token time(ms): 939.1155242919922, None-first token time(ms): 11074.248313903809, Non-first token Throughout(Token/s): 1.7156920687921857, Throughout(Token/s): 1.664812642767977