faiss::ascend::AscendIndexInt8Flat 多卡崩溃 aclopExecWithHandle: [Check][handle]param must not be null.
收藏回复举报
faiss::ascend::AscendIndexInt8Flat 多卡崩溃 aclopExecWithHandle: [Check][handle]param must not be null.
发表于2026-05-27 16:46:58
0 查看
[DEBUG] ASCENDCL(114,SearchEngineMainProgram):2026-05-27-15:19:00.828.106 [memory.cpp:257]287 aclrtMemcpy: start to execute aclrtMemcpy, destMaxSize = 256, srcSize = 256, kind = 1

[DEBUG] RUNTIME(114,SearchEngineMainProgram):2026-05-27-15:19:00.828.114 [logger.cc:542] 287 MemCopySync: memcpy sync, dst=0x1241c0084400, src=0xf85a18003250, count=256, kind=1

[DEBUG] DRV(114,SearchEngineMainProgram):2026-05-27-15:19:00.828.123 [ascend][curpid: 114, 287][drv][devmm][drvMemcpy 455]Argument. (dst=0x1241c0084400; src=0xf85a18003250; ByteCount=256)

[INFO] ASCENDCL(114,SearchEngineMainProgram):2026-05-27-15:19:00.828.186 [op.cpp:176]287 aclopExecWithHandle: start to execute aclopExecWithHandle

[ERROR] ASCENDCL(114,SearchEngineMainProgram):2026-05-27-15:19:00.828.236 [op.cpp:177]287 aclopExecWithHandle: [Check][handle]param must not be null.

[INFO] GE(114,SearchEngineMainProgram):2026-05-27-15:19:00.828.261 [error_manager.cc:298]287 ReportErrMessage:report error_message, error_code:EH0002, work_stream_id:11400287.

[ERROR] APP(114,SearchEngineMainProgram):2026-05-27-15:19:00.828.448 [log_inner.cpp:77]287 exec:ascenddaemon/utils/AscendOperator.cpp:107: "Error: 'err == ACL_ERROR_NONE' failed"

[ERROR] APP(114,SearchEngineMainProgram):2026-05-27-15:19:00.828.458 [log_inner.cpp:77]287 exec:ascenddaemon/utils/AscendOperator.cpp:107: "Error: 'err == ACL_ERROR_NONE' failed: ACL error 100000"

terminate called after throwing an instance of 'ascend::AscendException'

  what():  Error in void ascend::AscendOperator::exec(std::vector<const aclDataBuffer*>&, std::vector<aclDataBuffer*>&, aclrtStream) at ascenddaemon/utils/AscendOperator.cpp:107: Error: 'err == ACL_ERROR_NONE' failed: ACL error 100000

测试环境:atlas 300vPro 

驱动版本:

# npu-smi info

+--------------------------------------------------------------------------------------------------------+

| npu-smi 22.0.3                                   Version: 22.0.3                                       |

+-------------------------------+-----------------+------------------------------------------------------+

| NPU     Name                  | Health          | Power(W)     Temp(C)           Hugepages-Usage(page) |

| Chip    Device                | Bus-Id          | AICore(%)    Memory-Usage(MB)                        |

+===============================+=================+======================================================+

进程配置的2张卡:6,8

docker容器内部卡:0,1

算子库:1_TopkFlat_1_2_-1_-1_-1_1_2_-1_-1_-1_8_2_-1_-1_-1_7_2_-1_-1_-1_9_2_-1_1_2_-1_-1_9_2_-1_-1.om

每张卡独立初始化:new faiss::ascend::AscendIndexInt8Flat(dim, (faiss::MetricType)METHOD, conf_int8);

操作步骤:每张卡创建句柄,然后删除卡1,对卡0调用search接口崩溃。

详细日志见附件。

求助mxIndex模块专家分析一下崩溃原因。

我要发帖子