minikube: [--driver=docker --container-runtime=cri-o] kube-proxy fails with apply caps: operation not permitted

What Happened?

I’m running minikube on windows 11 with WSL2 Ubuntu 20.04 The kube-proxy pod cannot start, due to an error " kubelet Error: container create failed: time=“2021-12-30T17:58:21Z” level=error msg=“container_linux.go:380: starting container process caused: apply caps: operation not permitted”

This error was already reported in #12705. But I have always the same issue. I tried to be on 9th november commit but I have always the same issue

Steps to reproduce the issue


$ make local-kicbase
$ minikube start --driver=docker --container-runtime=cri-o --base-image="local/kicbase:latest"
$ kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS                 RESTARTS        AGE
kube-system   coredns-78fcd69978-wfd7p           0/1     ContainerCreating      0               13m
kube-system   etcd-minikube                      1/1     Running                0               14m
kube-system   kindnet-flvtt                      1/1     Running                5 (2m44s ago)   13m
kube-system   kube-apiserver-minikube            1/1     Running                0               14m
kube-system   kube-controller-manager-minikube   1/1     Running                0               14m
kube-system   kube-proxy-mpjw5                   0/1     CreateContainerError   0               13m
kube-system   kube-scheduler-minikube            1/1     Running                0               14m
kube-system   storage-provisioner                1/1     Running                2 (30s ago)   14m

kubectl -n kube-system describe pod kube-proxy-mpjw5

Name:                 kube-proxy-mpjw5
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 minikube/192.168.49.2
Start Time:           Thu, 30 Dec 2021 18:58:20 +0100
Labels:               controller-revision-hash=674d79d6f9
                      k8s-app=kube-proxy
                      pod-template-generation=1
Annotations:          <none>
Status:               Pending
IP:                   192.168.49.2
IPs:
  IP:           192.168.49.2
Controlled By:  DaemonSet/kube-proxy
Containers:
  kube-proxy:
    Container ID:
    Image:         k8s.gcr.io/kube-proxy:v1.22.3
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/local/bin/kube-proxy
      --config=/var/lib/kube-proxy/config.conf
      --hostname-override=$(NODE_NAME)
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/kube-proxy from kube-proxy (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2s856 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-proxy:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-proxy
    Optional:  false
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  kube-api-access-2s856:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  16m                 default-scheduler  Successfully assigned kube-system/kube-proxy-mpjw5 to minikube
  Warning  Failed     16m                 kubelet            Error: container create failed: time="2021-12-30T17:58:21Z" level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
  Warning  Failed     16m                 kubelet            Error: container create failed: time="2021-12-30T17:58:23Z" level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
  Warning  Failed     16m                 kubelet            Error: container create failed: time="2021-12-30T17:58:36Z" level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
  Warning  Failed     15m                 kubelet            Error: container create failed: time="2021-12-30T17:58:50Z" level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
  Warning  Failed     15m                 kubelet            Error: container create failed: time="2021-12-30T17:59:04Z" level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
  Warning  Failed     15m                 kubelet            Error: container create failed: time="2021-12-30T17:59:19Z" level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
  Warning  Failed     15m                 kubelet            Error: container create failed: time="2021-12-30T17:59:32Z" level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
  Warning  Failed     14m                 kubelet            Error: container create failed: time="2021-12-30T17:59:45Z" level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
  Warning  Failed     14m                 kubelet            Error: container create failed: time="2021-12-30T17:59:59Z" level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
  Warning  Failed     14m (x3 over 14m)   kubelet            (combined from similar events): Error: container create failed: time="2021-12-30T18:00:42Z" level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
  Normal   Pulled     79s (x69 over 16m)  kubelet            Container image "k8s.gcr.io/kube-proxy:v1.22.3" already present on machine

docker info

``` Client: Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc., v0.7.1) compose: Docker Compose (Docker Inc., v2.2.1) scan: Docker Scan (Docker Inc., v0.14.0) WARNING: Plugin "/usr/libexec/docker/cli-plugins/docker-app" is not valid: failed to fetch metadata: fork/exec /usr/libexec/docker/cli-plugins/docker-app: no such file or directory

Server: Containers: 5 Running: 3 Paused: 0 Stopped: 2 Images: 16 Server Version: 20.10.11 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc version: v1.0.2-0-g52b36a2 init version: de40ad0 Security Options: seccomp Profile: default Kernel Version: 5.10.74.3-microsoft-standard-WSL2 Operating System: Docker Desktop OSType: linux Architecture: x86_64 CPUs: 12 Total Memory: 15.63GiB Name: docker-desktop ID: 3XKP:Y5MD:X6YH:7WGT:JBSV:K6H2:6BDI:5XOQ:APB6:XXF4:UP3J:54ZM Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

</p>
</details>

### Attach the log file

[log.txt](https://github.com/kubernetes/minikube/files/7794162/log.txt)


### Operating System

Other

### Driver

Docker

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 17 (6 by maintainers)

Most upvoted comments

I had a similar issue but was able to make it work as explained here. I was on Linux Desktop but maybe the steps to verify can be used.

Hi @jbdamiano, did the above comment help? Are you still working on this issue?

klaases on Oct 6, 2022