镜像为mindie:1.0.RC3-800I-A2-x86_64
配置文件如下：
{
"Version": "1.0.0",
"LogConfig": {
"logLevel": "Info",
"logFileSize": 20,
"logFileNum": 20,
"logPath": "logs/mindservice.log"
},
"ServerConfig": {
"ipAddress": "127.0.0.1",
"managementIpAddress": "127.0.0.2",
"port": 18080,
"managementPort": 1026,
"metricsPort": 1027,
"allowAllZeroIpListening": false,
"maxLinkNum": 1000,
"httpsEnabled": false,
"fullTextEnabled": false,
"tlsCaPath": "security/ca/",
"tlsCaFile": [
"ca.pem"
],
"tlsCert": "security/certs/server.pem",
"tlsPk": "security/keys/server.key.pem",
"tlsPkPwd": "security/pass/key_pwd.txt",
"tlsCrl": "security/certs/server_crl.pem",
"managementTlsCaFile": [
"management_ca.pem"
],
"managementTlsCert": "security/certs/management/server.pem",
"managementTlsPk": "security/keys/management/server.key.pem",
"managementTlsPkPwd": "security/pass/management/key_pwd.txt",
"managementTlsCrl": "security/certs/management/server_crl.pem",
"kmcKsfMaster": "tools/pmt/master/ksfa",
"kmcKsfStandby": "tools/pmt/standby/ksfb",
"inferMode": "standard",
"interCommTLSEnabled": true,
"interCommPort": 1121,
"interCommTlsCaFile": "security/grpc/ca/ca.pem",
"interCommTlsCert": "security/grpc/certs/server.pem",
"interCommPk": "security/grpc/keys/server.key.pem",
"interCommPkPwd": "security/grpc/pass/key_pwd.txt",
"interCommTlsCrl": "security/grpc/certs/server_crl.pem",
"openAiSupport": "vllm"
},
"BackendConfig": {
"backendName": "mindieservice_llm_engine",
"modelInstanceNumber": 1,
"npuDeviceIds" : [[0,1,2,3,4,5,6,7]],
"tokenizerProcessNumber": 8,
"multiNodesInferEnabled": false,
"multiNodesInferPort": 1120,
"interNodeTLSEnabled": true,
"interNodeTlsCaFile": "security/grpc/ca/ca.pem",
"interNodeTlsCert": "security/grpc/certs/server.pem",
"interNodeTlsPk": "security/grpc/keys/server.key.pem",
"interNodeTlsPkPwd": "security/grpc/pass/mindie_server_key_pwd.txt",
"interNodeTlsCrl": "security/grpc/certs/server_crl.pem",
"interNodeKmcKsfMaster": "tools/pmt/master/ksfa",
"interNodeKmcKsfStandby": "tools/pmt/standby/ksfb",
"ModelDeployConfig": {
"maxSeqLen": 2560,
"maxInputTokenLen": 2048,
"truncation": false,
"ModelConfig": [
{
"modelInstanceType": "Standard",
"modelName": "qwen1.5-32b-chat",
"modelWeightPath": "/path-to-weights/Qwen1.5-32b-chat",
"worldSize": 8,
"cpuMemSize": 5,
"npuMemSize": 8,
"backendType": "atb"
}
]
},
"ScheduleConfig": {
"templateType": "Standard",
"templateName": "Standard_LLM",
"cacheBlockSize": 128,
"maxPrefillBatchSize": 50,
"maxPrefillTokens": 8192,
"prefillTimeMsPerReq": 150,
"prefillPolicyType": 0,
"decodeTimeMsPerReq": 50,
"decodePolicyType": 0,
"maxBatchSize": 200,
"maxIterTimes": 512,
"maxPreemptCount": 0,
"supportSelectBatch": false,
"maxQueueDelayMicroseconds": 5000
}
}
}
卡信息如下：
启动llm-service报错：
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
2025-03-06 09:53:26,795 [INFO] [pid: 27333] cpu_binding.py-205: rank_id: 0, device_id: 0, numa_id: 0, shard_devices: [0, 1, 2, 3], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
2025-03-06 09:53:26,797 [INFO] [pid: 27333] cpu_binding.py-231: process 27333, new_affinity is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], cpu count 10
2025-03-06 09:53:26,909 [INFO] [pid: 27339] cpu_binding.py-205: rank_id: 3, device_id: 3, numa_id: 0, shard_devices: [0, 1, 2, 3], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
2025-03-06 09:53:26,911 [INFO] [pid: 27339] cpu_binding.py-231: process 27339, new_affinity is [50, 51, 52, 53, 54, 55, 56, 57, 58, 59], cpu count 10
2025-03-06 09:53:26,911 [INFO] [pid: 27346] cpu_binding.py-205: rank_id: 4, device_id: 4, numa_id: 1, shard_devices: [4, 5, 6, 7], cpus: [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
2025-03-06 09:53:26,913 [INFO] [pid: 27346] cpu_binding.py-231: process 27346, new_affinity is [20, 21, 22, 23, 24, 25, 26, 27, 28, 29], cpu count 10
2025-03-06 09:53:27,064 [INFO] [pid: 27360] cpu_binding.py-205: rank_id: 6, device_id: 6, numa_id: 1, shard_devices: [4, 5, 6, 7], cpus: [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
2025-03-06 09:53:27,065 [INFO] [pid: 27333] logging.py-180: model_runner.quantize: None
, model_runner.kv_quant_type: None
, model_runner.dytpe: torch.bfloat16
2025-03-06 09:53:27,065 [INFO] [pid: 27360] cpu_binding.py-231: process 27360, new_affinity is [60, 61, 62, 63, 64, 65, 66, 67, 68, 69], cpu count 10
2025-03-06 09:53:27,155 [INFO] [pid: 27353] cpu_binding.py-205: rank_id: 5, device_id: 5, numa_id: 1, shard_devices: [4, 5, 6, 7], cpus: [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
2025-03-06 09:53:27,157 [INFO] [pid: 27353] cpu_binding.py-231: process 27353, new_affinity is [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], cpu count 10
2025-03-06 09:53:27,362 [INFO] [pid: 27335] cpu_binding.py-205: rank_id: 1, device_id: 1, numa_id: 0, shard_devices: [0, 1, 2, 3], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
2025-03-06 09:53:27,363 [INFO] [pid: 27335] cpu_binding.py-231: process 27335, new_affinity is [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], cpu count 10
2025-03-06 09:53:27,401 [INFO] [pid: 27337] cpu_binding.py-205: rank_id: 2, device_id: 2, numa_id: 0, shard_devices: [0, 1, 2, 3], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
2025-03-06 09:53:27,403 [INFO] [pid: 27337] cpu_binding.py-231: process 27337, new_affinity is [40, 41, 42, 43, 44, 45, 46, 47, 48, 49], cpu count 10
2025-03-06 09:53:27,411 [INFO] [pid: 27367] cpu_binding.py-205: rank_id: 7, device_id: 7, numa_id: 1, shard_devices: [4, 5, 6, 7], cpus: [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
2025-03-06 09:53:27,413 [INFO] [pid: 27367] cpu_binding.py-231: process 27367, new_affinity is [70, 71, 72, 73, 74, 75, 76, 77, 78, 79], cpu count 10
2025-03-06 09:53:32,116 [INFO] [pid: 27333] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:32,116 [INFO] [pid: 27333] logging.py-180: init tokenizer done: Qwen2TokenizerFast(name_or_path='/path-to-weights/Qwen1.5-32b-chat', vocab_size=151643, model_max_length=32768, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additional_special_tokens': ['<|im_start|>', '<|im_end|>']}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
2025-03-06 09:53:32,125 [INFO] [pid: 27333] logging.py-180: NPUSocInfo(soc_name='', soc_version=202, need_nz=True, matmul_nd_nz=False)
2025-03-06 09:53:32,161 [INFO] [pid: 27346] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:32,293 [INFO] [pid: 27333] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:32,350 [INFO] [pid: 27339] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:32,366 [INFO] [pid: 27346] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:32,664 [INFO] [pid: 27339] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:34,098 [INFO] [pid: 27360] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:34,311 [INFO] [pid: 27360] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:34,622 [INFO] [pid: 27353] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:34,821 [INFO] [pid: 27353] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:36,377 [INFO] [pid: 27367] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:36,484 [INFO] [pid: 27337] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:36,621 [INFO] [pid: 27367] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:36,676 [INFO] [pid: 27337] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:37,075 [INFO] [pid: 27335] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:37,234 [INFO] [pid: 27335] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:48,796 [INFO] [pid: 27333] logging.py-180: model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-63): 64 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
2025-03-06 09:53:49,461 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27339] 2025-03-06-09:53:49.454.154 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:49 (PID:27339, Device:3, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27339] 2025-03-06-09:53:49.454.154 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:49 (PID:27339, Device:3, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:49,464 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:49,694 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27333] 2025-03-06-09:53:49.687.121 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:49 (PID:27333, Device:0, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27333] 2025-03-06-09:53:49.687.121 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:49 (PID:27333, Device:0, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:49,698 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:55,696 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27337] 2025-03-06-09:53:55.688.239 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:55 (PID:27337, Device:2, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27337] 2025-03-06-09:53:55.688.239 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:55 (PID:27337, Device:2, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:55,701 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56,216 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27360] 2025-03-06-09:53:56.209.546 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27360, Device:6, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27360] 2025-03-06-09:53:56.209.546 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27360, Device:6, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:56,219 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56,220 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27335] 2025-03-06-09:53:56.213.216 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27335, Device:1, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27335] 2025-03-06-09:53:56.213.216 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27335, Device:1, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:56,224 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56,518 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27353] 2025-03-06-09:53:56.509.554 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27353, Device:5, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27353] 2025-03-06-09:53:56.509.554 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27353, Device:5, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:56,521 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56,669 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27346] 2025-03-06-09:53:56.662.388 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27346, Device:4, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27346] 2025-03-06-09:53:56.662.388 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27346, Device:4, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:56,673 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56,795 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27367] 2025-03-06-09:53:56.787.969 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27367, Device:7, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27367] 2025-03-06-09:53:56.787.969 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27367, Device:7, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:56,798 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56.851452 27261 error [llm_infer_model_instance.cpp:45] Initialize modelBackends_ failed.
2025-03-06 09:53:56.851602 27261 error [llm_infer_model.cpp:20] Init model instance failed.
LLMInferEngine failed to init LLMInferModels
ERR: Failed to init endpoint! Please check the service log or console output.
Killed
点赞
收藏
回复
分享
举报
已解决
镜像为mindie:1.0.RC3-800I-A2-x86_64
配置文件如下：
{
"Version": "1.0.0",
"LogConfig": {
"logLevel": "Info",
"logFileSize": 20,
"logFileNum": 20,
"logPath": "logs/mindservice.log"
},
"ServerConfig": {
"ipAddress": "127.0.0.1",
"managementIpAddress": "127.0.0.2",
"port": 18080,
"managementPort": 1026,
"metricsPort": 1027,
"allowAllZeroIpListening": false,
"maxLinkNum": 1000,
"httpsEnabled": false,
"fullTextEnabled": false,
"tlsCaPath": "security/ca/",
"tlsCaFile": [
"ca.pem"
],
"tlsCert": "security/certs/server.pem",
"tlsPk": "security/keys/server.key.pem",
"tlsPkPwd": "security/pass/key_pwd.txt",
"tlsCrl": "security/certs/server_crl.pem",
"managementTlsCaFile": [
"management_ca.pem"
],
"managementTlsCert": "security/certs/management/server.pem",
"managementTlsPk": "security/keys/management/server.key.pem",
"managementTlsPkPwd": "security/pass/management/key_pwd.txt",
"managementTlsCrl": "security/certs/management/server_crl.pem",
"kmcKsfMaster": "tools/pmt/master/ksfa",
"kmcKsfStandby": "tools/pmt/standby/ksfb",
"inferMode": "standard",
"interCommTLSEnabled": true,
"interCommPort": 1121,
"interCommTlsCaFile": "security/grpc/ca/ca.pem",
"interCommTlsCert": "security/grpc/certs/server.pem",
"interCommPk": "security/grpc/keys/server.key.pem",
"interCommPkPwd": "security/grpc/pass/key_pwd.txt",
"interCommTlsCrl": "security/grpc/certs/server_crl.pem",
"openAiSupport": "vllm"
},
"BackendConfig": {
"backendName": "mindieservice_llm_engine",
"modelInstanceNumber": 1,
"npuDeviceIds" : [[0,1,2,3,4,5,6,7]],
"tokenizerProcessNumber": 8,
"multiNodesInferEnabled": false,
"multiNodesInferPort": 1120,
"interNodeTLSEnabled": true,
"interNodeTlsCaFile": "security/grpc/ca/ca.pem",
"interNodeTlsCert": "security/grpc/certs/server.pem",
"interNodeTlsPk": "security/grpc/keys/server.key.pem",
"interNodeTlsPkPwd": "security/grpc/pass/mindie_server_key_pwd.txt",
"interNodeTlsCrl": "security/grpc/certs/server_crl.pem",
"interNodeKmcKsfMaster": "tools/pmt/master/ksfa",
"interNodeKmcKsfStandby": "tools/pmt/standby/ksfb",
"ModelDeployConfig": {
"maxSeqLen": 2560,
"maxInputTokenLen": 2048,
"truncation": false,
"ModelConfig": [
{
"modelInstanceType": "Standard",
"modelName": "qwen1.5-32b-chat",
"modelWeightPath": "/path-to-weights/Qwen1.5-32b-chat",
"worldSize": 8,
"cpuMemSize": 5,
"npuMemSize": 8,
"backendType": "atb"
}
]
},
"ScheduleConfig": {
"templateType": "Standard",
"templateName": "Standard_LLM",
"cacheBlockSize": 128,
"maxPrefillBatchSize": 50,
"maxPrefillTokens": 8192,
"prefillTimeMsPerReq": 150,
"prefillPolicyType": 0,
"decodeTimeMsPerReq": 50,
"decodePolicyType": 0,
"maxBatchSize": 200,
"maxIterTimes": 512,
"maxPreemptCount": 0,
"supportSelectBatch": false,
"maxQueueDelayMicroseconds": 5000
}
}
}
卡信息如下：
启动llm-service报错：
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
/usr/local/python3.11.10/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
2025-03-06 09:53:26,795 [INFO] [pid: 27333] cpu_binding.py-205: rank_id: 0, device_id: 0, numa_id: 0, shard_devices: [0, 1, 2, 3], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
2025-03-06 09:53:26,797 [INFO] [pid: 27333] cpu_binding.py-231: process 27333, new_affinity is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], cpu count 10
2025-03-06 09:53:26,909 [INFO] [pid: 27339] cpu_binding.py-205: rank_id: 3, device_id: 3, numa_id: 0, shard_devices: [0, 1, 2, 3], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
2025-03-06 09:53:26,911 [INFO] [pid: 27339] cpu_binding.py-231: process 27339, new_affinity is [50, 51, 52, 53, 54, 55, 56, 57, 58, 59], cpu count 10
2025-03-06 09:53:26,911 [INFO] [pid: 27346] cpu_binding.py-205: rank_id: 4, device_id: 4, numa_id: 1, shard_devices: [4, 5, 6, 7], cpus: [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
2025-03-06 09:53:26,913 [INFO] [pid: 27346] cpu_binding.py-231: process 27346, new_affinity is [20, 21, 22, 23, 24, 25, 26, 27, 28, 29], cpu count 10
2025-03-06 09:53:27,064 [INFO] [pid: 27360] cpu_binding.py-205: rank_id: 6, device_id: 6, numa_id: 1, shard_devices: [4, 5, 6, 7], cpus: [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
2025-03-06 09:53:27,065 [INFO] [pid: 27333] logging.py-180: model_runner.quantize: None
, model_runner.kv_quant_type: None
, model_runner.dytpe: torch.bfloat16
2025-03-06 09:53:27,065 [INFO] [pid: 27360] cpu_binding.py-231: process 27360, new_affinity is [60, 61, 62, 63, 64, 65, 66, 67, 68, 69], cpu count 10
2025-03-06 09:53:27,155 [INFO] [pid: 27353] cpu_binding.py-205: rank_id: 5, device_id: 5, numa_id: 1, shard_devices: [4, 5, 6, 7], cpus: [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
2025-03-06 09:53:27,157 [INFO] [pid: 27353] cpu_binding.py-231: process 27353, new_affinity is [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], cpu count 10
2025-03-06 09:53:27,362 [INFO] [pid: 27335] cpu_binding.py-205: rank_id: 1, device_id: 1, numa_id: 0, shard_devices: [0, 1, 2, 3], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
2025-03-06 09:53:27,363 [INFO] [pid: 27335] cpu_binding.py-231: process 27335, new_affinity is [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], cpu count 10
2025-03-06 09:53:27,401 [INFO] [pid: 27337] cpu_binding.py-205: rank_id: 2, device_id: 2, numa_id: 0, shard_devices: [0, 1, 2, 3], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
2025-03-06 09:53:27,403 [INFO] [pid: 27337] cpu_binding.py-231: process 27337, new_affinity is [40, 41, 42, 43, 44, 45, 46, 47, 48, 49], cpu count 10
2025-03-06 09:53:27,411 [INFO] [pid: 27367] cpu_binding.py-205: rank_id: 7, device_id: 7, numa_id: 1, shard_devices: [4, 5, 6, 7], cpus: [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
2025-03-06 09:53:27,413 [INFO] [pid: 27367] cpu_binding.py-231: process 27367, new_affinity is [70, 71, 72, 73, 74, 75, 76, 77, 78, 79], cpu count 10
2025-03-06 09:53:32,116 [INFO] [pid: 27333] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:32,116 [INFO] [pid: 27333] logging.py-180: init tokenizer done: Qwen2TokenizerFast(name_or_path='/path-to-weights/Qwen1.5-32b-chat', vocab_size=151643, model_max_length=32768, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additional_special_tokens': ['<|im_start|>', '<|im_end|>']}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
2025-03-06 09:53:32,125 [INFO] [pid: 27333] logging.py-180: NPUSocInfo(soc_name='', soc_version=202, need_nz=True, matmul_nd_nz=False)
2025-03-06 09:53:32,161 [INFO] [pid: 27346] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:32,293 [INFO] [pid: 27333] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:32,350 [INFO] [pid: 27339] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:32,366 [INFO] [pid: 27346] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:32,664 [INFO] [pid: 27339] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:34,098 [INFO] [pid: 27360] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:34,311 [INFO] [pid: 27360] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:34,622 [INFO] [pid: 27353] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:34,821 [INFO] [pid: 27353] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:36,377 [INFO] [pid: 27367] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:36,484 [INFO] [pid: 27337] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:36,621 [INFO] [pid: 27367] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:36,676 [INFO] [pid: 27337] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:37,075 [INFO] [pid: 27335] dist.py-79: initialize_distributed has been Set
2025-03-06 09:53:37,234 [INFO] [pid: 27335] flash_causal_qwen2.py-103: >>>> qwen_DecoderModel is called.
2025-03-06 09:53:48,796 [INFO] [pid: 27333] logging.py-180: model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-63): 64 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
2025-03-06 09:53:49,461 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27339] 2025-03-06-09:53:49.454.154 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:49 (PID:27339, Device:3, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27339] 2025-03-06-09:53:49.454.154 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:49 (PID:27339, Device:3, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:49,464 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:49,694 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27333] 2025-03-06-09:53:49.687.121 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:49 (PID:27333, Device:0, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27333] 2025-03-06-09:53:49.687.121 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:49 (PID:27333, Device:0, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:49,698 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:55,696 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27337] 2025-03-06-09:53:55.688.239 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:55 (PID:27337, Device:2, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27337] 2025-03-06-09:53:55.688.239 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:55 (PID:27337, Device:2, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:55,701 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56,216 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27360] 2025-03-06-09:53:56.209.546 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27360, Device:6, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27360] 2025-03-06-09:53:56.209.546 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27360, Device:6, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:56,219 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56,220 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27335] 2025-03-06-09:53:56.213.216 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27335, Device:1, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27335] 2025-03-06-09:53:56.213.216 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27335, Device:1, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:56,224 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56,518 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27353] 2025-03-06-09:53:56.509.554 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27353, Device:5, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27353] 2025-03-06-09:53:56.509.554 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27353, Device:5, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:56,521 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56,669 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27346] 2025-03-06-09:53:56.662.388 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27346, Device:4, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27346] 2025-03-06-09:53:56.662.388 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27346, Device:4, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:56,673 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56,795 [ERROR] model.py:39 - [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27367] 2025-03-06-09:53:56.787.969 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27367, Device:7, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 144, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 101, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 245, in warm_up
self.generator_backend.warm_up(model_inputs)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 128, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 92, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 65, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 92, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 157, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 380, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/models/qwen2/flash_causal_qwen2.py", line 120, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/llm_model/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 27367] 2025-03-06-09:53:56.787.969 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-03-06-09:53:56 (PID:27367, Device:7, RankID:-1) ERR01100 OPS call acl api failed
2025-03-06 09:53:56,798 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-03-06 09:53:56.851452 27261 error [llm_infer_model_instance.cpp:45] Initialize modelBackends_ failed.
2025-03-06 09:53:56.851602 27261 error [llm_infer_model.cpp:20] Init model instance failed.
LLMInferEngine failed to init LLMInferModels
ERR: Failed to init endpoint! Please check the service log or console output.
Killed