多个embedding模型运行在不同芯片上报错
收藏回复举报
多个embedding模型运行在不同芯片上报错
t('forum.solved') 已解决
发表于2025-01-22 17:12:32
0 查看
我使用mindie1.0里的/usr/local/Ascend/atb-models/examples/models/embedding/ 

里提供的三个模型,我在一个进程里把他们三个模型都加载进同一个芯片可以正常运行,但是如果后起的模型加载到另一个芯片里就会报错,此时之前加载的模型也不能正常运行了,这个是正常的吗?有啥办法解决吗?

报错信息为:

 

[2025-01-22 13:57:00.669327] [error] [107052] [rt_backend_module.cpp:180] rt KernelLaunch With Handle fail, error:107003 

[2025-01-22 13:57:00.669763] [error] [107052] [kernel_base.cpp:307] Mki RtFunction LaunchWithHandle fail 

[2025-01-22 13:57:00.669788] [error] [107052] [mki_node_implement.cpp:211] GatherOpsRunner_25_0[0] GatherF16Int64Kernel run fail, launchParam 

:specificParam: {"axis":[0],"batchDims":0} 

intensors[0]: dtype:float16, format:nd, dims:[250002, 1024], dataSize:512004096, data:1 

intensors[1]: dtype:int64, format:nd, dims:[1, 512], dataSize:4096, data:1 

outtensors[0]: dtype:float16, format:nd, dims:[1, 512, 1024], dataSize:1048576, data:1 

 

st: code:7, msg:Mki RtFunction LaunchWithHandle fail 

[2025-01-22 13:57:00.669894] [error] [107052] [ops_runner.cpp:602] RunKernel failed! ret:3 

[2025-01-22 13:57:00.669908] [error] [107052] [ops_runner.cpp:397] RunAllKernel failed! ret:3 

[2025-01-22 13:57:00.669923] [error] [107052] [runner.cpp:115] GatherOpsRunner_25_0:1 Execute Failed. st: 3 

[2025-01-22 13:57:00.669940] [error] [107052] [graph_runner.cpp:906] EmbeddingLayerRunner_25:0  node[0] execute fail, runner name:GatherOpsRunner 

[2025-01-22 13:57:00.669955] [error] [107052] [runner.cpp:115] EmbeddingLayerRunner_25:1 Execute Failed. st: 3 

[2025-01-22 13:57:00.669967] [error] [107052] [operation_base.cpp:726] EmbeddingLayer_25 execute EmbeddingLayerRunner fail 

[2025-01-22 13:57:00.669978] [error] [107052] [operation_base.cpp:764] EmbeddingLayer_25 execute fail, error code: 3 

[2025-01-22 13:57:00.670281] [error] [107052] [rt_backend_module.cpp:180] rt KernelLaunch With Handle fail, error:107003 

[2025-01-22 13:57:00.670678] [error] [107052] [kernel_base.cpp:307] Mki RtFunction LaunchWithHandle fail 

[2025-01-22 13:57:00.670699] [error] [107052] [mki_node_implement.cpp:211] LinearOpsRunner_26_0_0[0] TransdataNdToNzKernel run fail, launchParam 

:specificParam: {"outCrops":[0,0],"transdataType":2} 

intensors[0]: dtype:float16, format:nd, dims:[1, 512, 1024], dataSize:1048576, data:1 

outtensors 

我要发帖子