Using k8s kind "rootlessly" without Docker

Posted on September 3, 2021 • 3 minutes • 623 words

So you probably already heard the news Docker Desktop is no longer free . While this mostly affect macOS and Windows users and I use Pop!_OS, I still would like to see if we can get by without Docker at all.

I’ve been using nerdctl for quite awhile now and while nerdctl mostly fill my needs for docker CLI, I “kinda” need kind CLI to create test cluster for testing purpose. However kind still needs docker.

What if I alias nerdctl to docker? I did that and then try again

ln -s nerdctl docker
kind create cluster --name test

Now I’m getting different error.

ERROR: failed to create cluster: running kind with rootless provider requires cgroup v2, see https://kind.sigs.k8s.io/docs/user/rootless/

Well, this is good right? I just have to enable cgroup v2 and then I should be good to go? Usually I do have cgroup v2 enable but I’m trying Pop!_OS at the moment and the kernel is kinda old. So I upgrade kernel to the latest stable (5.13), using a custom kernel by Xanmod.

uname -a
Linux x300 5.13.14-xanmod1 #0~git20210903.d548864 SMP PREEMPT Fri Sep 3 13:21:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

The steps are a bit different when I was using Manjaro but it basically boil down

Adding a new kernel parameter systemd.unified_cgroup_hierarchy=1. The instruction on kind sig page doesn’t work for me. On Pop!_OS, I need to use kernelstub
Delegate a few more controllers, namely cpu, cpuset and io.

sudo kernelstub -a "systemd.unified_cgroup_hierarchy=1"
sudo reboot

After that, I’m getting a tiny bit different error

ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

This looks like, because only memory and pids controllers are delegated to non-root users but we need more, specially cpu, cpuset and io controllers.

We can verify this by, the following command. You will see only memory and pids are delegated.

cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers
memory pids

You can delegate more by doing this, and verify with the above command.

# mkdir -p /etc/systemd/system/[email protected]
# cat > /etc/systemd/system/[email protected]/delegate.conf << EOF
[Service]
Delegate=cpu cpuset io memory pids
EOF
# systemctl daemon-reload

If all is good, this is what you see

cat /sys/fs/cgroup/user.slice/user-(id -u).slice/user@(id -u).service/cgroup.controllers
cpuset cpu io memory pids

I thought it should be ok now but no, I still got the above error

ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

At this point, I decided to jump into kind codebase to see the condition that trigger that error. Turns out, they use docker info command to see if cgroup v2 is active and to see what kind of controllers got delegated. And nerdctl doesn’t emit those info yet.

nerdctl info looks like below and the docker one has a lot more information regarding where cpushare is supported, pid is supported, etc…

{
  "ID": "86232191-2d46-475b-be0c-1472c5174763",
  "Driver": "overlayfs",
  "Plugins": {
    "Log": [
      "json-file"
    ],
    "Storage": [
      "native",
      "overlayfs"
    ]
  },
  "LoggingDriver": "json-file",
  "CgroupDriver": "systemd",
  "CgroupVersion": "2",
  "KernelVersion": "5.13.14-xanmod1",
  "OperatingSystem": "Pop!_OS 21.04",
  "OSType": "linux",
  "Architecture": "x86_64",
  "Name": "x300",
  "ServerVersion": "v1.5.5",
  "SecurityOptions": [
    "name=seccomp,profile=default",
    "name=cgroupns",
    "name=rootless"
  ]
}

So at this point, I can only log the issue on <code>nerdctl</code> repo and see if it’s really the only problem or there would be sth else that prevent kind working with nerdctl.

Update: So I tried to fix nerdctl info command and once I did, I got another error regarding nerdctl ps where --filter flag is not yet implemented. So thí is where I stopped for now. I will revisit this later.

ERROR: failed to create cluster: failed to list clusters: command "docker ps -a --filter label=io.x-k8s.kind.cluster=test --format '{{.Names}}'" failed with error: exit status 1
Command Output: Incorrect Usage: flag provided but not defined: -filter