昇腾社区首页
支持
消息
我的论坛主页
论坛首页
/
NPU调用相关容器服务在用docker-compose设置restart:always自启后，对应容器开机自启动失败原因
收藏回复举报
NPU调用相关容器服务在用docker-compose设置restart:always自启后，对应容器开机自启动失败原因
t('forum.solved') 已解决
新人帖
发表于2024-02-02 16:04:37
0 查看

NPU为Atlas 200 AI 加速模块，以EP模式运行,

OS: ubuntu20.04.6,

安装一下固件驱动和CANN 及ascend docker runtime支持

Atlas-200-npu-firmware_6.0.0.run

Atlas-200-npu-driver_6.0.0_linux-x86_64.run

Ascend-cann-nnrt_6.3.RC2.alpha002_linux-x86_64.run

Ascend-docker-runtime_5.0.RC1_linux-x86_64.run

通过docker compose设置autorestart之后，重启设备，发现对应npu调用容器启动失败，对应其他容器随docker服务自启动正常，失败后可以正常通过手动启动npu调用容器

部分系统日志可以看到17:31:25附近是有对应容器启动失败的报错，

Feb  1 17:31:24 zdlc-desktop dockerd[1311]: time="2024-02-01T17:31:24.724108172+08:00" level=info msg="Starting up"
...
Feb  1 17:31:25 zdlc-desktop containerd[793]: time="2024-02-01T17:31:25.663163780+08:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  1 17:31:25 zdlc-desktop containerd[793]: time="2024-02-01T17:31:25.663485582+08:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/f0de65b5c0c905373f9e0194c908ecf386029ed7879ac02f1d44f695294e94c8 pid=1939 runtime=io.containerd.runc.v2
Feb  1 17:31:25 zdlc-desktop systemd[889]: run-docker-runtime\x2drunc-moby-444bff3ddc4d68d0a3d24c04c2916eb7f30e4126972df437d6cf53bb59842e6b-runc.XGWmkx.mount: Succeeded.
Feb  1 17:31:25 zdlc-desktop containerd[793]: time="2024-02-01T17:31:25.684079009+08:00" level=info msg="shim disconnected" id=f0de65b5c0c905373f9e0194c908ecf386029ed7879ac02f1d44f695294e94c8
Feb  1 17:31:25 zdlc-desktop containerd[793]: time="2024-02-01T17:31:25.684124443+08:00" level=warning msg="cleaning up after shim disconnected" id=f0de65b5c0c905373f9e0194c908ecf386029ed7879ac02f1d44f695294e94c8 namespace=moby
Feb  1 17:31:25 zdlc-desktop containerd[793]: time="2024-02-01T17:31:25.684134750+08:00" level=info msg="cleaning up dead shim"
Feb  1 17:31:25 zdlc-desktop kernel: [    8.618199] eth0: renamed from veth9ca91b0
Feb  1 17:31:25 zdlc-desktop containerd[793]: time="2024-02-01T17:31:25.694063030+08:00" level=warning msg="cleanup warnings time=\"2024-02-01T17:31:25+08:00\" level=info msg=\"starting signal loop\" namespace=moby pid=1994 runtime=io.containerd.runc.v2\ntime=\"2024-02-01T17:31:25+08:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/containerd/io.containerd.runtime.v2.task/moby/f0de65b5c0c905373f9e0194c908ecf386029ed7879ac02f1d44f695294e94c8/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n"
Feb  1 17:31:25 zdlc-desktop containerd[793]: time="2024-02-01T17:31:25.694318021+08:00" level=error msg="copy shim log" error="read /proc/self/fd/27: file already closed"
Feb  1 17:31:25 zdlc-desktop dockerd[1311]: time="2024-02-01T17:31:25.694720803+08:00" level=error msg="stream copy error: reading from a closed fifo"
Feb  1 17:31:25 zdlc-desktop dockerd[1311]: time="2024-02-01T17:31:25.695209653+08:00" level=error msg="stream copy error: reading from a closed fifo"
Feb  1 17:31:25 zdlc-desktop kernel: [    8.645635] IPv6: ADDRCONF(NETDEV_CHANGE): vethc76ad1d: link becomes ready
Feb  1 17:31:25 zdlc-desktop kernel: [    8.645669] br-daba872822e1: port 1(vethc76ad1d) entered blocking state
.....
Feb  1 17:31:25 zdlc-desktop systemd[1]: var-lib-docker-overlay2-db70f19f84ade65f5ae36019c86e6d1408b405ce0d2bb7f6349e659338b2f13b-merged.mount: Succeeded.
Feb  1 17:31:25 zdlc-desktop systemd[889]: var-lib-docker-overlay2-db70f19f84ade65f5ae36019c86e6d1408b405ce0d2bb7f6349e659338b2f13b-merged.mount: Succeeded.
Feb  1 17:31:25 zdlc-desktop dockerd[1311]: time="2024-02-01T17:31:25.907346925+08:00" level=error msg="failed to start container" container=f0de65b5c0c905373f9e0194c908ecf386029ed7879ac02f1d44f695294e94c8 error="failed to create task for container: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/moby/f0de65b5c0c905373f9e0194c908ecf386029ed7879ac02f1d44f695294e94c8/log.json: no such file or directory): /usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime did not terminate successfully: exit status 1: unknown"
Feb  1 17:31:25 zdlc-desktop systemd-udevd[367]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  1 17:31:25 zdlc-desktop systemd-udevd[392]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  1 17:31:26 zdlc-desktop kernel: [    8.934138] eth0: renamed from veth4a0c8b5
Feb  1 17:31:26 zdlc-desktop systemd-udevd[367]: veth1d2dfac: Could not generate persistent MAC: No data available
Feb  1 17:31:26 zdlc-desktop systemd-udevd[392]: vethf69c479: Could not generate persistent MAC: No data available

联系device_boot_init.log有：

[aicpu] [2024-02-01 16:03:55] device boot init start, physical id: 0, ip_addr: 127.0.0.1, user: root
[aicpu] [2024-02-01 16:08:01] <<<<<<<< device_0 boot init >>>>>>>>
[aicpu] [2024-02-01 16:08:01] device boot init start, physical id: 0, ip_addr: 127.0.0.1, user: root
[aicpu] [2024-02-01 17:32:42] <<<<<<<< device_0 boot init >>>>>>>>
[aicpu] [2024-02-01 17:32:42] device boot init start, physical id: 0, ip_addr: 127.0.0.1, user: root

我要发帖子

我们使用cookie来确保您的高速浏览体验。继续浏览本站，即表示您同意我们使用cookie。查看详情