边云协同分布式安全训练-U-shape切分,配置--num-layer-list 参数,报错IndexError: pop from empty list
收藏回复举报
边云协同分布式安全训练-U-shape切分,配置--num-layer-list 参数,报错IndexError: pop from empty list
t('forum.solved') 已解决
新人帖
发表于2026-04-10 17:03:23
0 查看

按照https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/docs/zh/pytorch/training/finetune/mcore/layerwise_disaggregated_training.md指导文档部署边云协同分布式安全训练。

在examples/mcore/qwen3/ckpt_convert_qwen3_hf2mcore.sh脚本中,添加--num-layer-list参数后,执行脚本会上报如下错误:

Traceback (most recent call last):

  File "/MindSpeed-LLM/MindSpeed-LLM/convert_ckpt_v2.py", line 104, in <module>

    main()

  File "/MindSpeed-LLM/MindSpeed-LLM/mindspeed_llm/training/utils.py", line 1106, in wrapper

    return func(*args, **kwargs)

  File "/MindSpeed-LLM/MindSpeed-LLM/convert_ckpt_v2.py", line 91, in main

    converter = Hf2MgConvert(args)

  File "/MindSpeed-LLM/MindSpeed-LLM/mindspeed_llm/tasks/checkpoint/convert_hf2mg.py", line 76, in __init__

    self.get_pprank_hf_layeridxs()

  File "/MindSpeed-LLM/MindSpeed-LLM/mindspeed_llm/tasks/checkpoint/convert_hf2mg.py", line 164, in get_pprank_hf_layeridxs

    self.pprank_layer_idxs[pp_rank] = [num_layer_list_.pop(0) for _ in range(layers_each_pp[pp_rank])]

  File "/MindSpeed-LLM/MindSpeed-LLM/mindspeed_llm/tasks/checkpoint/convert_hf2mg.py", line 164, in <listcomp>

    self.pprank_layer_idxs[pp_rank] = [num_layer_list_.pop(0) for _ in range(layers_each_pp[pp_rank])]

IndexError: pop from empty list

[ERROR] 2026-04-10-16:39:19 (PID:16057, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

删除--num-layer-list 16,32,16 # U-shape切分后,权重转换脚本可正常运行

我要发帖子