在昇腾300I Pro使用BGE-Rerank-v2-m3推理特别慢,Currently High Accuracy but Low Performance OP with 64-bit has been used
收藏回复举报
在昇腾300I Pro使用BGE-Rerank-v2-m3推理特别慢,Currently High Accuracy but Low Performance OP with 64-bit has been used
t('forum.solved') 已解决
发表于2024-11-28 15:41:52
0 查看

驱动版本:23.0.0,CANN版本:8.0.RC2

模型定义:

from FlagEmbedding import FlagReranker

reranker = FlagReranker(model_name_or_path=rerank_models[RERANK_MODEL], use_fp16=True, device="npu")

第一次推理会出现warning:

[W1128 14:52:35.391381143 compiler_depend.ts:45] Warning: The oprator of ne is executed, Currently High Accuracy but Low Performance OP with 64-bit has been used, Please Do Some Cast at Python Functions with 32-bit for Better Performance! (function operator())

每次推理都特别慢,并且会出现Warning: tiling offset out of range, index: 32:

cke_3558.png

我要发帖子