你好,
我在训练百川33B模型,遇到dp, mp, pp设置问题导致模型编译出错能否支持解决下,
如下设置会导致:OM错误, 如下截图
# default parallel of device num = 16 for Atlas 800
parallel_config:
data_parallel: 8
model_parallel: 4
pipeline_stage: 1
micro_batch_num: 8
vocab_emb_dp: True
gradient_aggregation_group: 4
# when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process.
micro_batch_interleave_num: 1

当改变设置如下后, 提示新的错误如下图
# default parallel of device num = 16 for Atlas 800
parallel_config:
data_parallel: 2
model_parallel: 4
pipeline_stage: 4
micro_batch_num: 8
vocab_emb_dp: True
gradient_aggregation_group: 4
# when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process.
micro_batch_interleave_num: 1

请帮忙看下,如何解决训练问题,谢谢。
你好,
我在训练百川33B模型,遇到dp, mp, pp设置问题导致模型编译出错能否支持解决下,
如下设置会导致:OM错误, 如下截图
# default parallel of device num = 16 for Atlas 800
parallel_config:
data_parallel: 8
model_parallel: 4
pipeline_stage: 1
micro_batch_num: 8
vocab_emb_dp: True
gradient_aggregation_group: 4
# when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process.
micro_batch_interleave_num: 1
当改变设置如下后, 提示新的错误如下图
# default parallel of device num = 16 for Atlas 800
parallel_config:
data_parallel: 2
model_parallel: 4
pipeline_stage: 4
micro_batch_num: 8
vocab_emb_dp: True
gradient_aggregation_group: 4
# when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process.
micro_batch_interleave_num: 1
请帮忙看下,如何解决训练问题,谢谢。