测试了3个模型,每个模型单独能运行成功(使用4卡、2卡、2卡NPU),然后我想试一下同时跑3个模型,修改mindIE config.json运行后提示:
the size of npuDeviceIds (subset) does not equal to worldSize
设置里主要修改:
"modelInstanceNumber" : 3,
"npuDeviceIds" : [[0,1,2,3],[4,5],[6,7]],
"ModelConfig": [
{
"modelInstanceType": "Standard",
"modelName": "Qwen2.5-72B-Instruct",
"modelWeightPath": "/storage/Qwen/Qwen2.5-72B-Instruct",
"worldSize": 4,
"cpuMemSize": 10,
"npuMemSize": -1,
"backendType": "atb",
"trustRemoteCode": true
},
{
"modelInstanceType": "Standard",
"modelName": "DeepSeek-R1-Distill-Qwen-32B",
"modelWeightPath": "/storage/Qwen/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
"worldSize": 2,
"cpuMemSize": 5,
"npuMemSize": -1,
"backendType": "atb",
"trustRemoteCode": true
},
{
"modelInstanceType": "Standard",
"modelName": "QwQ-32B",
"modelWeightPath": "/storage/Qwen/QwQ-32B",
"worldSize": 2,
"cpuMemSize": 5,
"npuMemSize": -1,
"backendType": "atb",
"trustRemoteCode": true
}
]
},
我的理解大概是可以推理3个模型,4+2+2,为啥不能运行?除了分成3个docker外,有比较好的解决方案吗?
顺便,如果显存占用不满的话,能否用同一张卡运行embedding模型?embedding模型资源占用较少,单卡感觉浪费。
测试了3个模型,每个模型单独能运行成功(使用4卡、2卡、2卡NPU),然后我想试一下同时跑3个模型,修改mindIE config.json运行后提示:
the size of npuDeviceIds (subset) does not equal to worldSize
设置里主要修改:
"modelInstanceNumber" : 3,
"npuDeviceIds" : [[0,1,2,3],[4,5],[6,7]],
"ModelConfig": [
{
"modelInstanceType": "Standard",
"modelName": "Qwen2.5-72B-Instruct",
"modelWeightPath": "/storage/Qwen/Qwen2.5-72B-Instruct",
"worldSize": 4,
"cpuMemSize": 10,
"npuMemSize": -1,
"backendType": "atb",
"trustRemoteCode": true
},
{
"modelInstanceType": "Standard",
"modelName": "DeepSeek-R1-Distill-Qwen-32B",
"modelWeightPath": "/storage/Qwen/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
"worldSize": 2,
"cpuMemSize": 5,
"npuMemSize": -1,
"backendType": "atb",
"trustRemoteCode": true
},
{
"modelInstanceType": "Standard",
"modelName": "QwQ-32B",
"modelWeightPath": "/storage/Qwen/QwQ-32B",
"worldSize": 2,
"cpuMemSize": 5,
"npuMemSize": -1,
"backendType": "atb",
"trustRemoteCode": true
}
]
},
我的理解大概是可以推理3个模型,4+2+2,为啥不能运行?除了分成3个docker外,有比较好的解决方案吗?
顺便,如果显存占用不满的话,能否用同一张卡运行embedding模型?embedding模型资源占用较少,单卡感觉浪费。