昇腾论坛

如题，是用300I-Duo-mindie-2.2.rc1的镜像，修改了config文件中的maxPrefillTokens等参数，执行以下命令

export HCCL_DETERMINISTIC=false

export HCCL_OP_EXPANSION_MODE="AIV"

export NPU_MEMORY_FRACTION=0.96

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True

export OMP_NUM_THREADS=1

export LD_LIBRARY_PATH=/usr/local/Ascend/atb-models/lib:$LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/usr/local/lib64/python3.11/site-packages/torch/lib:$LD_LIBRARY_PATH

cd /usr/local/Ascend/mindie/latest/mindie-service/

./bin/mindieservice_daemon

出现以下显示

g_mainPid = 219897

LogLevelDynamicHandler start

ConfigManager: Load Config from /usr/local/Ascend/mindie/2.2.RC1/mindie-service/conf/config.json.

The configName speculationGamma is not found, use default value.

[ConfigManager::ExecuteConfigInteractions] Configuration interactions completed successfully

[ConfigManager::InitConfigManager] Successfully init config manager

LogLevelDynamicHandler start

[W compiler_depend.ts:62] Warning: Cannot create tensor with NZ format while dim < 2, tensor will be created with ND format. (function operator())

LogLevelDynamicHandler start

Daemon start success!

但是在使用以下调用命令时，服务终端出现显存不够的问题，

curl 127.0.0.1:1025/v1/chat/completions -d ' {

"model": "qwen2.5_vl_3b",

"messages": [{

"role": "user",

"content": [

{"type": "image_url","image_url": "/workspace/data/test.png"},

{"type": "text", "text": "Explain the details in the image."}

]

}],

"max_tokens": 10,

"stream": false,

"do_sample":true,

"repetition_penalty": 1.00,

"temperature": 0.6,

"top_p": 0.8,

"top_k": 20

错误日志：

LogLevelDynamicHandler start

Daemon start success!

/usr/local/lib64/python3.11/site-packages/torchvision/transforms/functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True).

warnings.warn(

Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/request_router.py", line 23, in run

self._target(*self._args, **self._kwargs)

File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/request_router.py", line 94, in do_inference

self.router_impl.execute(execute_request)

File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/router_impl.py", line 209, in execute

self._generate(execute_request, is_prefill=True, is_mix=False)

File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/router_impl.py", line 406, in _generate

generate_output = self._handle_requests(input_metadata_composite)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/router_impl.py", line 526, in _handle_requests

generate_output = self.generator.generate_token(metadata)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 376, in generate_token

raise e

File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 355, in generate_token

generation_output = self.plugin.generate_token(input_metadata)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 91, in wrapper

return func(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/plugins/plugin_manager.py", line 115, in generate_token

result = self.generator_backend.forward(model_inputs, q_lens=self.plugin_data_param.q_len,

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 69, in wrapper

return func(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 222, in forward

logits = self._forward(model_inputs, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 721, in _forward

logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 130, in forward

result = self.forward_from_model_inputs(model_inputs, npu_cache, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 208, in forward_from_model_inputs

result = self.forward_tensor(

^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 280, in forward_tensor

raise e

File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 266, in forward_tensor

result = self.model_runner.forward(

^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 324, in forward

res = self.model.forward(**kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 118, in forward

inputs_embeds, image_grid_thw, video_grid_thw, second_per_grid_ts = self.prepare_prefill_token_service(

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 197, in prepare_prefill_token_service

image_features = self.vision_tower(image_pixel.to(self.vision_tower.dtype), image_grid_thw)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl

return self._call_impl(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl

return forward_call(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vit_atb.py", line 812, in forward

vision_features = self.encoder(

^^^^^^^^^^^^^

File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl

return self._call_impl(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl

return forward_call(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vit_atb.py", line 661, in forward

hidden_states = self.graph.forward(self.graph_inputs, self.graph_outputs, self.graph_param)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: NPU out of memory. Tried to allocate 602.00 MiB (NPU 0; 21.02 GiB total capacity; 18.40 GiB already allocated; 18.40 GiB current active; 776.18 MiB free; 18.46 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '

/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 7 leaked shared_memory objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '

/usr/lib64/python3.11/multiprocessing/resource_tracker.py:267: UserWarning: resource_tracker: '//219897_0_shared_sync_link_request': [Errno 2] No such file or directory: '//219897_0_shared_sync_link_request'