昇腾社区首页
支持
消息
我的论坛主页
论坛首页
/
服务化推理启动失败：resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
收藏回复举报
服务化推理启动失败：resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
t('forum.solved') 已解决
发表于2025-03-17 13:40:19
0 查看

环境

atlas 800T A2训练 服务器 环境：

使用docker方式启动服务化推理

宿主机：

系统：openeurle22.03TLS 驱动： 24.1.rc2

docker镜像：

镜像文件名称：mindie_2.0.T3_800IA2.tar 系统：24.03TLS cann： 8.0.t63 mindle： 2.0.T3 python版本：3.11.6

使用docker的方式启动服务化推理出现如下错误：

Arguments: ()
[2025-03-17 13:28:31,963] [9049] [281472599781728] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-03-17 13:28:32,168] [9020] [281472181006688] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-03-17 13:28:32,695] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=223, need_nz=False, matmul_nd_nz=False)
[2025-03-17 13:28:32,836] [9034] [281471866040672] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-03-17 13:28:33,404] [9018] [281472538767712] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set

[2025-03-17 13:33:32,658] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : model:
 FlashDeepseekv2ForCausalLM(
  (rotary_embedding): DeepseekV2YarnRotaryEmbedding()
  (attn_mask): AttentionMask()
  (model): FlashDeepseekV2Model(
    (embed_tokens): TensorEmbedding()
    (layers): ModuleList(
      (0-2): 3 x FlashDeepseekV2DecoderLayer(
        (self_attn): FlashDeepseekV2Attention(
          (q_a_proj): TensorReplicatedLinear(
            (linear): W8A8LinearStatic()
          )
          (q_b_proj): TensorParallelColumnLinear(
            (linear): W8A8LinearStatic()
          )
          (kv_a_proj_with_mqa): TensorReplicatedLinear(
            (linear): W8A8LinearStatic()
          )
          (kv_a_layernorm): DeepseekV2RMSNorm()
          (kv_b_proj): TensorParallelColumnLinear(
            (linear): FastLinear()
          )
          (o_proj): TensorParallelRowLinear(
            (linear): W8A8LinearStatic()
          )
          (rotary_emb): DeepseekV2YarnRotaryEmbedding()
          (q_a_layernorm): DeepseekV2RMSNormBias()
        )
        (mlp): DeepseekV2MLP(
          (act): SiLU()
          (gate_up_proj): TensorParallelColumnLinear(
            (linear): W8A8DynamicLinearStatic()
          )
          (down_proj): TensorParallelRowLinear(
            (linear): W8A8DynamicLinearStatic()
          )
          (act_fn): SiLU()
        )
        (input_layernorm): DeepseekV2RMSNormBias()
        (post_attention_layernorm): DeepseekV2RMSNorm()
      )
      (3-60): 58 x FlashDeepseekV2DecoderLayer(
        (self_attn): FlashDeepseekV2Attention(
          (q_a_proj): TensorReplicatedLinear(
            (linear): W8A8LinearStatic()
          )
          (q_b_proj): TensorParallelColumnLinear(
            (linear): W8A8LinearStatic()
          )
          (kv_a_proj_with_mqa): TensorReplicatedLinear(
            (linear): W8A8LinearStatic()
          )
          (kv_a_layernorm): DeepseekV2RMSNorm()
          (kv_b_proj): TensorParallelColumnLinear(
            (linear): FastLinear()
          )
          (o_proj): TensorParallelRowLinear(
            (linear): W8A8LinearStatic()
          )
          (rotary_emb): DeepseekV2YarnRotaryEmbedding()
          (q_a_layernorm): DeepseekV2RMSNormBias()
        )
        (mlp): DeepseekV2MoE(
          (gate): FastLinear()
          (gate_up_proj): ModuleList(
            (0-255): 256 x TensorParallelColumnLinear(
              (linear): W8A8DynamicLinearStatic()
            )
          )
          (down_proj): ModuleList(
            (0-255): 256 x TensorParallelRowLinear(
              (linear): W8A8DynamicLinearStatic()
            )
          )
          (shared_experts): DeepseekV2MLP(
            (act): SiLU()
            (gate_up_proj): TensorParallelColumnLinear(
              (linear): W8A8DynamicLinearStatic()
            )
            (down_proj): TensorParallelRowLinear(
              (linear): W8A8DynamicLinearStatic()
            )
            (act_fn): SiLU()
          )
        )
        (input_layernorm): DeepseekV2RMSNormBias()
        (post_attention_layernorm): DeepseekV2RMSNorm()
      )
    )
    (norm): DeepseekV2RMSNorm()
  )
  (lm_head): TensorParallelHead(
    (linear): FastLinear()
  )
  (embed_tokens): TensorEmbedding()
)
[2025-03-17 13:33:33,111] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:33,145] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:34,968] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:34,984] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,081] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,098] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,197] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,214] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,312] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,328] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,428] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,444] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,542] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,558] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,657] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,673] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,774] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,790] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,888] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:35,905] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,002] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,018] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,117] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,134] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,233] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,250] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,350] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,367] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,465] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,482] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,584] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,601] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,702] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,719] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,817] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,834] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,933] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:36,949] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,047] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,064] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,164] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,181] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,281] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,298] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,398] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,415] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,516] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,533] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,632] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,650] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,750] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,766] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,865] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,881] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,979] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:37,996] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,099] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,115] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,241] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,258] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,401] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,418] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,560] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,577] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,677] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,694] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,794] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,811] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,911] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:38,928] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,030] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,047] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,149] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,167] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,270] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,287] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,390] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,407] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,508] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,525] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,626] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,647] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,749] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,766] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,877] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:39,899] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,011] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,033] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,142] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,164] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,276] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,298] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,423] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,446] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,558] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,580] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,688] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,713] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,821] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,842] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,952] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:40,973] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,086] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,108] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,218] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,240] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,349] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,372] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,474] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,495] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,602] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,624] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,730] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,755] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,880] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:41,907] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-17 13:33:42,296] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 128, 1, 576])
[2025-03-17 13:33:42,297] [9016] [281472742060384] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281469772853712 id of vcache is 281469772853616
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

我要发帖子

我们使用cookie来确保您的高速浏览体验。继续浏览本站，即表示您同意我们使用cookie。查看详情