4台 910b2 服务器运行 deepseek-r1(1.3TB) 起服务时报错 integrated_gmmRunner_5_4_9_0_2:0 node[0] SetupNodeRunners fail, error code:12
收藏回复举报
4台 910b2 服务器运行 deepseek-r1(1.3TB) 起服务时报错 integrated_gmmRunner_5_4_9_0_2:0 node[0] SetupNodeRunners fail, error code:12
t('forum.solved') 已解决
新人帖
发表于2025-03-06 14:20:50
0 查看

部署参考教程: https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/MindIE/LLM/DeepSeek/DeepSeek-R1

报错截图一

大致意思是在"模型加载预热" 时挂了

cke_45284.png

报错截图二 (这里不清楚怎么解决)

说是在 prefill 时 setup fail, error code: 12

cke_33424.png

硬件环境(4台机器规格都一致):

0b3bd8dd-3367-495d-ba92-b1f7efffba01.png

已安装驱动&固件等版本(自认为已经很新了):

Ascend-hdk-910b-npu-driver_24.1.rc3_linux-aarch64.run

Ascend-hdk-910b-npu-firmware_7.5.0.1.129.run

Ascend-cann-toolkit_8.0.RC3.alpha003_linux-aarch64.run

Ascend-cann-kernels-910b_8.0.RC3.alpha003_linux-aarch64.run

Ascend-docker-runtime_6.0.RC3_linux-aarch64.run

rank_table_file.json 配置(感觉也没啥问题):

cke_112283.png

目前 config.json 配置 (已经尽力不爆显存了):

{
    "Version" : "1.0.0",
    "LogConfig" :
    {
        "logLevel" : "Info",
        "logFileSize" : 20,
        "logFileNum" : 20,
        "logPath" : "logs/mindie-server.log"
    },

    "ServerConfig" :
    {
        "ipAddress" : "169.254.45.87",
        "managementIpAddress" : "127.0.0.2",
        "port" : 1025,
        "managementPort" : 1026,
        "metricsPort" : 1027,
        "allowAllZeroIpListening" : false,
        "maxLinkNum" : 300,
        "httpsEnabled": false,
        "fullTextEnabled" : false,
        "tlsCaPath" : "security/ca/",
        "tlsCaFile" : ["ca.pem"],
        "tlsCert" : "security/certs/server.pem",
        "tlsPk" : "security/keys/server.key.pem",
        "tlsPkPwd" : "security/pass/key_pwd.txt",
        "tlsCrlPath" : "security/certs/",
        "tlsCrlFiles" : ["server_crl.pem"],
        "managementTlsCaFile" : ["management_ca.pem"],
        "managementTlsCert" : "security/certs/management/server.pem",
        "managementTlsPk" : "security/keys/management/server.key.pem",
        "managementTlsPkPwd" : "security/pass/management/key_pwd.txt",
        "managementTlsCrlPath" : "security/management/certs/",
        "managementTlsCrlFiles" : ["server_crl.pem"],
        "kmcKsfMaster" : "tools/pmt/master/ksfa",
        "kmcKsfStandby" : "tools/pmt/standby/ksfb",
        "inferMode" : "standard",
        "interCommTLSEnabled": false,
        "interCommPort" : 1121,
        "interCommTlsCaPath" : "security/grpc/ca/",
        "interCommTlsCaFiles" : ["ca.pem"],
        "interCommTlsCert" : "security/grpc/certs/server.pem",
        "interCommPk" : "security/grpc/keys/server.key.pem",
        "interCommPkPwd" : "security/grpc/pass/key_pwd.txt",
        "interCommTlsCrlPath" : "security/grpc/certs/",
        "interCommTlsCrlFiles" : ["server_crl.pem"],
        "openAiSupport" : "vllm"
    },

    "BackendConfig" : {
        "backendName" : "mindieservice_llm_engine",
        "modelInstanceNumber" : 1,
        "npuDeviceIds" : [[0,1,2,3,4,5,6,7]],
        "tokenizerProcessNumber" : 8,
        "multiNodesInferEnabled": true,
        "multiNodesInferPort" : 1120,
        "interNodeTLSEnabled": false,
        "interNodeTlsCaPath" : "security/grpc/ca/",
        "interNodeTlsCaFiles" : ["ca.pem"],
        "interNodeTlsCert" : "security/grpc/certs/server.pem",
        "interNodeTlsPk" : "security/grpc/keys/server.key.pem",
        "interNodeTlsPkPwd" : "security/grpc/pass/mindie_server_key_pwd.txt",
        "interNodeTlsCrlPath" : "security/grpc/certs/",
        "interNodeTlsCrlFiles" : ["server_crl.pem"],
        "interNodeKmcKsfMaster" : "tools/pmt/master/ksfa",
        "interNodeKmcKsfStandby" : "tools/pmt/standby/ksfb",
        "ModelDeployConfig" :
        {
            "maxSeqLen" : 2048,
            "maxInputTokenLen" : 1024,
            "truncation" : true,
            "ModelConfig" : [
                {
                    "modelInstanceType" : "Standard",
                    "modelName": "DeepSeek-R1",
                    "modelWeightPath": "/data/models/DeepSeek-R1-BF16",
                    "worldSize" : 8,
                    "cpuMemSize" : 5,
                    "npuMemSize" : -1,
                    "backendType" : "atb",
                    "trustRemoteCode" : true
                }
            ]
        },

        "ScheduleConfig" :
        {
            "templateType" : "Standard",
            "templateName" : "Standard_LLM",
            "cacheBlockSize" : 128,
            "maxPrefillBatchSize" : 8,
            "maxPrefillTokens" : 1024,
            "prefillTimeMsPerReq" : 150,
            "prefillPolicyType" : 0,
            "decodeTimeMsPerReq" : 50,
            "decodePolicyType" : 0,
            "maxBatchSize" : 8,
            "maxIterTimes" : 512,
            "maxPreemptCount" : 0,
            "supportSelectBatch" : false,
            "maxQueueDelayMicroseconds" : 5000
        }
    }
}

重启大法已经试过了不行

哪怕是谁能提供一些解决方向也可以,感激不尽!

我要发帖子