MindIE版本：2.0.T3.1
执行命令：/usr/local/Ascend/mindie/latest/mindie-service/bin/mindieservice_daemon
报错：
[2025-04-22 07:21:30,144] [22065] [281463877333344] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=224, need_nz=False, matmul_nd_nz=False)
[2025-04-22 07:21:30,427] [22065] [281463877333344] [llm] [INFO] [dist.py-102] : ProcessGroupHCCL has been Set
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:30,532] [22081] [281464490815840] [llm] [INFO] [dist.py-116] : rank 3 init True, init_process_group has been activated
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:30,693] [22095] [281464791101792] [llm] [INFO] [dist.py-116] : rank 5 init True, init_process_group has been activated
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:30,888] [22102] [281464482361696] [llm] [INFO] [dist.py-116] : rank 6 init True, init_process_group has been activated
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:31,126] [22104] [281464172179808] [llm] [INFO] [dist.py-116] : rank 7 init True, init_process_group has been activated
[2025-04-22 07:21:31,125] [22067] [281464163725664] [llm] [INFO] [dist.py-116] : rank 1 init True, init_process_group has been activated
[2025-04-22 07:21:31,126] [22088] [281464037962080] [llm] [INFO] [dist.py-116] : rank 4 init True, init_process_group has been activated
[2025-04-22 07:21:31.022+0800] [22081] [281473510732128] [mindie-server] [ERROR] [model.py:40] : [Model] >>> Exception:weight vision_model.embeddings.class_embedding does not exist
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 38, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 93, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 85, in __init__
self.generator_backend = get_generator_backend(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/__init__.py", line 26, in get_generator_backend
return generator_cls(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 97, in __init__
super().__init__(model_config)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 113, in __init__
self.model_wrapper = get_model_wrapper(model_config, backend_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/__init__.py", line 15, in get_model_wrapper
return wrapper_cls(**model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 52, in __init__
self.model_runner.load_weights()
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 161, in load_weights
self.model = self.model_cls(self.config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 91, in __init__
self.init_vision_model()
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 170, in init_vision_model
self.init_module_weight(self.vision_model, self.weights, prefix="vision_model")
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 133, in init_module_weight
weights.get_tensor(f"{prefix}.{model_weight}"), requires_grad=False
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 153, in get_tensor
filename, tensor_name = self.get_filename(tensor_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 146, in get_filename
raise AssertionError(f"weight {tensor_name} does not exist")
AssertionError: weight vision_model.embeddings.class_embedding does not exist
[2025-04-22 07:21:31.022+0800] [22081] [281473510732128] [mindie-server] [ERROR] [model.py:43] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[2025-04-22 07:21:31.022+0800] [22095] [281473600975200] [mindie-server] [ERROR] [model.py:40] : [Model] >>> Exception:weight vision_model.embeddings.class_embedding does not exist
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 38, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 93, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 85, in __init__
self.generator_backend = get_generator_backend(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/__init__.py", line 26, in get_generator_backend
return generator_cls(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 97, in __init__
super().__init__(model_config)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 113, in __init__
self.model_wrapper = get_model_wrapper(model_config, backend_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/__init__.py", line 15, in get_model_wrapper
return wrapper_cls(**model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 52, in __init__
self.model_runner.load_weights()
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 161, in load_weights
self.model = self.model_cls(self.config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 91, in __init__
self.init_vision_model()
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 170, in init_vision_model
self.init_module_weight(self.vision_model, self.weights, prefix="vision_model")
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 133, in init_module_weight
weights.get_tensor(f"{prefix}.{model_weight}"), requires_grad=False
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 153, in get_tensor
filename, tensor_name = self.get_filename(tensor_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 146, in get_filename
raise AssertionError(f"weight {tensor_name} does not exist")
AssertionError: weight vision_model.embeddings.class_embedding does not exist
[2025-04-22 07:21:31.022+0800] [22095] [281473600975200] [mindie-server] [ERROR] [model.py:43] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[2025-04-22 07:21:31.022+0800] [22102] [281473357640032] [mindie-server] [ERROR] [model.py:40] : [Model] >>> Exception:weight vision_model.embeddings.class_embedding does not exist
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 38, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 93, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 85, in __init__
self.generator_backend = get_generator_backend(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/__init__.py", line 26, in get_generator_backend
return generator_cls(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 97, in __init__
super().__init__(model_config)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 113, in __init__
self.model_wrapper = get_model_wrapper(model_config, backend_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/__init__.py", line 15, in get_model_wrapper
return wrapper_cls(**model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 52, in __init__
self.model_runner.load_weights()
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 161, in load_weights
self.model = self.model_cls(self.config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 91, in __init__
self.init_vision_model()
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 170, in init_vision_model
self.init_module_weight(self.vision_model, self.weights, prefix="vision_model")
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 133, in init_module_weight
weights.get_tensor(f"{prefix}.{model_weight}"), requires_grad=False
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 153, in get_tensor
filename, tensor_name = self.get_filename(tensor_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 146, in get_filename
raise AssertionError(f"weight {tensor_name} does not exist")
AssertionError: weight vision_model.embeddings.class_embedding does not exist
[2025-04-22 07:21:31.022+0800] [22102] [281473357640032] [mindie-server] [ERROR] [model.py:43] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[2025-04-22 07:21:31.342+08:00] [21798] [21799] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.343+08:00] [21798] [21799] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22081, status 9
[2025-04-22 07:21:31.344+08:00] [21798] [21799] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22081 exit, Please check the service log or python log.
[2025-04-22 07:21:31.374+08:00] [21798] [21798] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.375+08:00] [21798] [21798] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22095, status 9
[2025-04-22 07:21:31.376+08:00] [21798] [21798] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22095 exit, Please check the service log or python log.
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:31,397] [22074] [281463970853216] [llm] [INFO] [dist.py-116] : rank 2 init True, init_process_group has been activated
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:31,403] [22065] [281463877333344] [llm] [INFO] [dist.py-116] : rank 0 init True, init_process_group has been activated
[2025-04-22 07:21:31.462+08:00] [21798] [21802] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.462+08:00] [21798] [21802] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22102, status 9
[2025-04-22 07:21:31.463+08:00] [21798] [21802] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22102 exit, Please check the service log or python log.
[2025-04-22 07:21:31.022+0800] [22088] [281473055650144] [mindie-server] [ERROR] [model.py:40] : [Model] >>> Exception:weight vision_model.embeddings.class_embedding does not exist
...........
AssertionError: weight vision_model.embeddings.class_embedding does not exist
[2025-04-22 07:21:31.022+0800] [22074] [281473120661856] [mindie-server] [ERROR] [model.py:43] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[2025-04-22 07:21:31.885+08:00] [21798] [21995] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.886+08:00] [21798] [21995] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22067, status 9
[2025-04-22 07:21:31.886+08:00] [21798] [21995] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22067 exit, Please check the service log or python log.
[2025-04-22 07:21:31.921+08:00] [21798] [21996] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.922+08:00] [21798] [21996] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22104, status 9
[2025-04-22 07:21:31.923+08:00] [21798] [21996] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22104 exit, Please check the service log or python log.
[2025-04-22 07:21:31.931+08:00] [21798] [21997] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.932+08:00] [21798] [21997] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22088, status 9
[2025-04-22 07:21:31.933+08:00] [21798] [21997] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22088 exit, Please check the service log or python log.
[2025-04-22 07:21:32.071+08:00] [21798] [22061] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:32.072+08:00] [21798] [22061] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22074, status 9
[2025-04-22 07:21:32.072+08:00] [21798] [22061] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22074 exit, Please check the service log or python log.
[2025-04-22 07:21:32.095+08:00] [21798] [22062] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:32.096+08:00] [21798] [22062] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22065, status 9
[2025-04-22 07:21:32.096+08:00] [21798] [22062] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22065 exit, Please check the service log or python log.
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
Process ForkServerProcess-6:
Process ForkServerProcess-4:
Process ForkServerProcess-9:
Process ForkServerProcess-8:
Process ForkServerProcess-5:
Process ForkServerProcess-7:
Process ForkServerProcess-2:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
During handling of the above exception, another exception occurred:
During handling of the above exception, another exception occurred:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
During handling of the above exception, another exception occurred:
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
Traceback (most recent call last):
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "<string>", line 2, in append
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "<string>", line 2, in append
File "<string>", line 2, in append
File "<string>", line 2, in append
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "<string>", line 2, in append
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
ConnectionRefusedError: [Errno 111] Connection refused
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
ConnectionRefusedError: [Errno 111] Connection refused
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "<string>", line 2, in append
ConnectionRefusedError: [Errno 111] Connection refused
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
Process ForkServerProcess-8:
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "<string>", line 2, in append
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...
点赞
收藏
回复
分享
举报
已解决
MindIE版本：2.0.T3.1
执行命令：/usr/local/Ascend/mindie/latest/mindie-service/bin/mindieservice_daemon
报错：
[2025-04-22 07:21:30,144] [22065] [281463877333344] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=224, need_nz=False, matmul_nd_nz=False)
[2025-04-22 07:21:30,427] [22065] [281463877333344] [llm] [INFO] [dist.py-102] : ProcessGroupHCCL has been Set
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:30,532] [22081] [281464490815840] [llm] [INFO] [dist.py-116] : rank 3 init True, init_process_group has been activated
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:30,693] [22095] [281464791101792] [llm] [INFO] [dist.py-116] : rank 5 init True, init_process_group has been activated
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:30,888] [22102] [281464482361696] [llm] [INFO] [dist.py-116] : rank 6 init True, init_process_group has been activated
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:31,126] [22104] [281464172179808] [llm] [INFO] [dist.py-116] : rank 7 init True, init_process_group has been activated
[2025-04-22 07:21:31,125] [22067] [281464163725664] [llm] [INFO] [dist.py-116] : rank 1 init True, init_process_group has been activated
[2025-04-22 07:21:31,126] [22088] [281464037962080] [llm] [INFO] [dist.py-116] : rank 4 init True, init_process_group has been activated
[2025-04-22 07:21:31.022+0800] [22081] [281473510732128] [mindie-server] [ERROR] [model.py:40] : [Model] >>> Exception:weight vision_model.embeddings.class_embedding does not exist
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 38, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 93, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 85, in __init__
self.generator_backend = get_generator_backend(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/__init__.py", line 26, in get_generator_backend
return generator_cls(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 97, in __init__
super().__init__(model_config)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 113, in __init__
self.model_wrapper = get_model_wrapper(model_config, backend_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/__init__.py", line 15, in get_model_wrapper
return wrapper_cls(**model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 52, in __init__
self.model_runner.load_weights()
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 161, in load_weights
self.model = self.model_cls(self.config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 91, in __init__
self.init_vision_model()
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 170, in init_vision_model
self.init_module_weight(self.vision_model, self.weights, prefix="vision_model")
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 133, in init_module_weight
weights.get_tensor(f"{prefix}.{model_weight}"), requires_grad=False
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 153, in get_tensor
filename, tensor_name = self.get_filename(tensor_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 146, in get_filename
raise AssertionError(f"weight {tensor_name} does not exist")
AssertionError: weight vision_model.embeddings.class_embedding does not exist
[2025-04-22 07:21:31.022+0800] [22081] [281473510732128] [mindie-server] [ERROR] [model.py:43] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[2025-04-22 07:21:31.022+0800] [22095] [281473600975200] [mindie-server] [ERROR] [model.py:40] : [Model] >>> Exception:weight vision_model.embeddings.class_embedding does not exist
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 38, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 93, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 85, in __init__
self.generator_backend = get_generator_backend(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/__init__.py", line 26, in get_generator_backend
return generator_cls(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 97, in __init__
super().__init__(model_config)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 113, in __init__
self.model_wrapper = get_model_wrapper(model_config, backend_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/__init__.py", line 15, in get_model_wrapper
return wrapper_cls(**model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 52, in __init__
self.model_runner.load_weights()
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 161, in load_weights
self.model = self.model_cls(self.config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 91, in __init__
self.init_vision_model()
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 170, in init_vision_model
self.init_module_weight(self.vision_model, self.weights, prefix="vision_model")
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 133, in init_module_weight
weights.get_tensor(f"{prefix}.{model_weight}"), requires_grad=False
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 153, in get_tensor
filename, tensor_name = self.get_filename(tensor_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 146, in get_filename
raise AssertionError(f"weight {tensor_name} does not exist")
AssertionError: weight vision_model.embeddings.class_embedding does not exist
[2025-04-22 07:21:31.022+0800] [22095] [281473600975200] [mindie-server] [ERROR] [model.py:43] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[2025-04-22 07:21:31.022+0800] [22102] [281473357640032] [mindie-server] [ERROR] [model.py:40] : [Model] >>> Exception:weight vision_model.embeddings.class_embedding does not exist
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 38, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 93, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 85, in __init__
self.generator_backend = get_generator_backend(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/__init__.py", line 26, in get_generator_backend
return generator_cls(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 97, in __init__
super().__init__(model_config)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 113, in __init__
self.model_wrapper = get_model_wrapper(model_config, backend_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/__init__.py", line 15, in get_model_wrapper
return wrapper_cls(**model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 52, in __init__
self.model_runner.load_weights()
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 161, in load_weights
self.model = self.model_cls(self.config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 91, in __init__
self.init_vision_model()
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 170, in init_vision_model
self.init_module_weight(self.vision_model, self.weights, prefix="vision_model")
File "/usr/local/Ascend/atb-models/atb_llm/models/internvl/flash_causal_internvl.py", line 133, in init_module_weight
weights.get_tensor(f"{prefix}.{model_weight}"), requires_grad=False
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 153, in get_tensor
filename, tensor_name = self.get_filename(tensor_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/weights.py", line 146, in get_filename
raise AssertionError(f"weight {tensor_name} does not exist")
AssertionError: weight vision_model.embeddings.class_embedding does not exist
[2025-04-22 07:21:31.022+0800] [22102] [281473357640032] [mindie-server] [ERROR] [model.py:43] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[2025-04-22 07:21:31.342+08:00] [21798] [21799] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.343+08:00] [21798] [21799] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22081, status 9
[2025-04-22 07:21:31.344+08:00] [21798] [21799] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22081 exit, Please check the service log or python log.
[2025-04-22 07:21:31.374+08:00] [21798] [21798] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.375+08:00] [21798] [21798] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22095, status 9
[2025-04-22 07:21:31.376+08:00] [21798] [21798] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22095 exit, Please check the service log or python log.
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:31,397] [22074] [281463970853216] [llm] [INFO] [dist.py-116] : rank 2 init True, init_process_group has been activated
[W compiler_depend.ts:795] Warning: The HCCL execution timeout 1800000ms is bigger than watchdog timeout 300000ms which is set by init_process_group! The plog may not be recorded. (function ProcessGroupHCCL)
[2025-04-22 07:21:31,403] [22065] [281463877333344] [llm] [INFO] [dist.py-116] : rank 0 init True, init_process_group has been activated
[2025-04-22 07:21:31.462+08:00] [21798] [21802] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.462+08:00] [21798] [21802] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22102, status 9
[2025-04-22 07:21:31.463+08:00] [21798] [21802] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22102 exit, Please check the service log or python log.
[2025-04-22 07:21:31.022+0800] [22088] [281473055650144] [mindie-server] [ERROR] [model.py:40] : [Model] >>> Exception:weight vision_model.embeddings.class_embedding does not exist
...........
AssertionError: weight vision_model.embeddings.class_embedding does not exist
[2025-04-22 07:21:31.022+0800] [22074] [281473120661856] [mindie-server] [ERROR] [model.py:43] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[2025-04-22 07:21:31.885+08:00] [21798] [21995] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.886+08:00] [21798] [21995] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22067, status 9
[2025-04-22 07:21:31.886+08:00] [21798] [21995] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22067 exit, Please check the service log or python log.
[2025-04-22 07:21:31.921+08:00] [21798] [21996] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.922+08:00] [21798] [21996] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22104, status 9
[2025-04-22 07:21:31.923+08:00] [21798] [21996] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22104 exit, Please check the service log or python log.
[2025-04-22 07:21:31.931+08:00] [21798] [21997] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:31.932+08:00] [21798] [21997] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22088, status 9
[2025-04-22 07:21:31.933+08:00] [21798] [21997] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22088 exit, Please check the service log or python log.
[2025-04-22 07:21:32.071+08:00] [21798] [22061] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:32.072+08:00] [21798] [22061] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22074, status 9
[2025-04-22 07:21:32.072+08:00] [21798] [22061] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22074 exit, Please check the service log or python log.
[2025-04-22 07:21:32.095+08:00] [21798] [22062] [mindie-server] [WARN] [llm_daemon.cpp:66] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-04-22 07:21:32.096+08:00] [21798] [22062] [mindie-server] [INFO] [llm_daemon.cpp:71] : [daemon] Daemon wait pid with 22065, status 9
[2025-04-22 07:21:32.096+08:00] [21798] [22062] [mindie-server] [ERROR] [llm_daemon.cpp:76] : [MIE04E010109] [daemon] Daemon wait pid with 22065 exit, Please check the service log or python log.
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
Process ForkServerProcess-6:
Process ForkServerProcess-4:
Process ForkServerProcess-9:
Process ForkServerProcess-8:
Process ForkServerProcess-5:
Process ForkServerProcess-7:
Process ForkServerProcess-2:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
During handling of the above exception, another exception occurred:
During handling of the above exception, another exception occurred:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
During handling of the above exception, another exception occurred:
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
Traceback (most recent call last):
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "<string>", line 2, in append
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "<string>", line 2, in append
File "<string>", line 2, in append
File "<string>", line 2, in append
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "<string>", line 2, in append
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
ConnectionRefusedError: [Errno 111] Connection refused
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
ConnectionRefusedError: [Errno 111] Connection refused
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "<string>", line 2, in append
ConnectionRefusedError: [Errno 111] Connection refused
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
Process ForkServerProcess-8:
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "<string>", line 2, in append
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...