我使用mindie1.0里的/usr/local/Ascend/atb-models/examples/models/embedding/
里提供的三个模型,我在一个进程里把他们三个模型都加载进同一个芯片可以正常运行,但是如果后起的模型加载到另一个芯片里就会报错,此时之前加载的模型也不能正常运行了,这个是正常的吗?有啥办法解决吗?
报错信息为:
[2025-01-22 13:57:00.669327] [error] [107052] [rt_backend_module.cpp:180] rt KernelLaunch With Handle fail, error:107003
[2025-01-22 13:57:00.669763] [error] [107052] [kernel_base.cpp:307] Mki RtFunction LaunchWithHandle fail
[2025-01-22 13:57:00.669788] [error] [107052] [mki_node_implement.cpp:211] GatherOpsRunner_25_0[0] GatherF16Int64Kernel run fail, launchParam
:specificParam: {"axis":[0],"batchDims":0}
intensors[0]: dtype:float16, format:nd, dims:[250002, 1024], dataSize:512004096, data:1
intensors[1]: dtype:int64, format:nd, dims:[1, 512], dataSize:4096, data:1
outtensors[0]: dtype:float16, format:nd, dims:[1, 512, 1024], dataSize:1048576, data:1
st: code:7, msg:Mki RtFunction LaunchWithHandle fail
[2025-01-22 13:57:00.669894] [error] [107052] [ops_runner.cpp:602] RunKernel failed! ret:3
[2025-01-22 13:57:00.669908] [error] [107052] [ops_runner.cpp:397] RunAllKernel failed! ret:3
[2025-01-22 13:57:00.669923] [error] [107052] [runner.cpp:115] GatherOpsRunner_25_0:1 Execute Failed. st: 3
[2025-01-22 13:57:00.669940] [error] [107052] [graph_runner.cpp:906] EmbeddingLayerRunner_25:0 node[0] execute fail, runner name:GatherOpsRunner
[2025-01-22 13:57:00.669955] [error] [107052] [runner.cpp:115] EmbeddingLayerRunner_25:1 Execute Failed. st: 3
[2025-01-22 13:57:00.669967] [error] [107052] [operation_base.cpp:726] EmbeddingLayer_25 execute EmbeddingLayerRunner fail
[2025-01-22 13:57:00.669978] [error] [107052] [operation_base.cpp:764] EmbeddingLayer_25 execute fail, error code: 3
[2025-01-22 13:57:00.670281] [error] [107052] [rt_backend_module.cpp:180] rt KernelLaunch With Handle fail, error:107003
[2025-01-22 13:57:00.670678] [error] [107052] [kernel_base.cpp:307] Mki RtFunction LaunchWithHandle fail
[2025-01-22 13:57:00.670699] [error] [107052] [mki_node_implement.cpp:211] LinearOpsRunner_26_0_0[0] TransdataNdToNzKernel run fail, launchParam
:specificParam: {"outCrops":[0,0],"transdataType":2}
intensors[0]: dtype:float16, format:nd, dims:[1, 512, 1024], dataSize:1048576, data:1
outtensors
里提供的三个模型,我在一个进程里把他们三个模型都加载进同一个芯片可以正常运行,但是如果后起的模型加载到另一个芯片里就会报错,此时之前加载的模型也不能正常运行了,这个是正常的吗?有啥办法解决吗?
报错信息为:
[2025-01-22 13:57:00.669327] [error] [107052] [rt_backend_module.cpp:180] rt KernelLaunch With Handle fail, error:107003
[2025-01-22 13:57:00.669763] [error] [107052] [kernel_base.cpp:307] Mki RtFunction LaunchWithHandle fail
[2025-01-22 13:57:00.669788] [error] [107052] [mki_node_implement.cpp:211] GatherOpsRunner_25_0[0] GatherF16Int64Kernel run fail, launchParam
:specificParam: {"axis":[0],"batchDims":0}
intensors[0]: dtype:float16, format:nd, dims:[250002, 1024], dataSize:512004096, data:1
intensors[1]: dtype:int64, format:nd, dims:[1, 512], dataSize:4096, data:1
outtensors[0]: dtype:float16, format:nd, dims:[1, 512, 1024], dataSize:1048576, data:1
st: code:7, msg:Mki RtFunction LaunchWithHandle fail
[2025-01-22 13:57:00.669894] [error] [107052] [ops_runner.cpp:602] RunKernel failed! ret:3
[2025-01-22 13:57:00.669908] [error] [107052] [ops_runner.cpp:397] RunAllKernel failed! ret:3
[2025-01-22 13:57:00.669923] [error] [107052] [runner.cpp:115] GatherOpsRunner_25_0:1 Execute Failed. st: 3
[2025-01-22 13:57:00.669940] [error] [107052] [graph_runner.cpp:906] EmbeddingLayerRunner_25:0 node[0] execute fail, runner name:GatherOpsRunner
[2025-01-22 13:57:00.669955] [error] [107052] [runner.cpp:115] EmbeddingLayerRunner_25:1 Execute Failed. st: 3
[2025-01-22 13:57:00.669967] [error] [107052] [operation_base.cpp:726] EmbeddingLayer_25 execute EmbeddingLayerRunner fail
[2025-01-22 13:57:00.669978] [error] [107052] [operation_base.cpp:764] EmbeddingLayer_25 execute fail, error code: 3
[2025-01-22 13:57:00.670281] [error] [107052] [rt_backend_module.cpp:180] rt KernelLaunch With Handle fail, error:107003
[2025-01-22 13:57:00.670678] [error] [107052] [kernel_base.cpp:307] Mki RtFunction LaunchWithHandle fail
[2025-01-22 13:57:00.670699] [error] [107052] [mki_node_implement.cpp:211] LinearOpsRunner_26_0_0[0] TransdataNdToNzKernel run fail, launchParam
:specificParam: {"outCrops":[0,0],"transdataType":2}
intensors[0]: dtype:float16, format:nd, dims:[1, 512, 1024], dataSize:1048576, data:1
outtensors