For one of our customers we needed to install the Docker-plugin for loki. To install it, you run:
docker plugin install loki
However this failed with the following error message:
Error response from daemon: dial unix /run/docker/plugins/d442cff8568254659f9aa7d2dd5a30526b69d1ac62593c628b65da8b9e933c6a/loki.sock: connect: no such file or directory
Running strace on the enable-command showed nothing really interesting:
[pid 40213] write(3, "POST /v1.41/plugins/loki/enable?"..., 157 <unfinished ...>
[pid 40207] <... nanosleep resumed>NULL) = 0
<snip>
[pid 40206] <... futex resumed>) = 0
[pid 40205] <... read resumed>"HTTP/1.1 500 Internal Server Err"..., 4096) = 377
[pid 40206] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
<snip>
[pid 40213] write(2, "Error response from daemon: dial"..., 169 <unfinished ...>
[pid 40207] futex(0xc000074848, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 40206] <... nanosleep resumed>NULL) = 0
Error response from daemon: dial unix /run/docker/plugins/d442cff8568254659f9aa7d2dd5a30526b69d1ac62593c628b65da8b9e933c6a/loki.sock: connect: no such file or directory
There was also an error in the logs:
Jul 23 13:12:52 egbr-abn-web01 containerd: time="2021-07-23T13:12:52.256314425+02:00" level=info msg="starting signal loop" namespace=plugins.moby path=/run/containerd/io.containerd.runtime.v2.task/plugins.moby/
d442cff8568254659f9aa7d2dd5a30526b69d1ac62593c628b65da8b9e933c6a pid=40222
Jul 23 13:12:52 egbr-abn-web01 dockerd: time="2021-07-23T13:12:52+02:00" level=error msg="standard_init_linux.go:228: exec user process caused: permission denied" plugin=d442cff8568254659f9aa7d2dd5a30526b69d1ac6
2593c628b65da8b9e933c6a
Permission denied is interesting! In the back of my head this made it click. As a security measure all our Linux servers have noexec set on the mounts for /var and /tmp. And noexec means no executables can be executed, but this is what Docker is trying to do here. Removing the noexec from the /var mount fixed the problem.