kind: [hyperkit minikube] NFS Provisioner going in crash loop

What happened: I used to use kind on docker desktop and was happily deploying nfs server in k8 using helm which is using image quay.io/kubernetes_incubator/nfs-provisioner:v2.3.0 now I have recently switched to minikube with no kubernetes to leverage the docker inside of it obviously because much resources were taken by the previous one, now when I am deploying the same nfs server on kind cluster with same config the pod for nfs provisioner goes into crash loop, now this happens on kind cluster in minikubes’ docker when I install nfs server, with no kubernetes, however, if i use minikube with kubernetes and deploy same NFS server using helm it runs successfully but as I try to submit a PVC it says it can’t provision the storage (even when the amount of storage i asked was 1Mi) i understand that could be something related to minikube but this issue is there, I am using minikube with hyperkit driver, I tried to debugg the issue to gather more information but couldn’t do it, so I would also like to request the direction I should go to debug this, or possibly make this work, also some hints on what might be happenig.

What you expected to happen: KinD cluster should be more agnostic towards different environments.

How to reproduce it (as minimally and precisely as possible): Step 1: Start minikube with no kubernetes minikube start --driver hyperkit --no-kubernetes; eval $(minikube -p minikube docker-env) Step 2: Create KinD cluster kind create cluster Step 3: Install NFS server (if you have helm else click here)

NFS yamls 

# Source: nfs-server-provisioner/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: nfs-server-provisioner
    chart: nfs-server-provisioner-1.1.3
    heritage: Helm
    release: nfs-provisioner
  name: nfs-provisioner-nfs-server-provisioner
---
# Source: nfs-server-provisioner/templates/storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: nfs
  labels:
    app: nfs-server-provisioner
    chart: nfs-server-provisioner-1.1.3
    heritage: Helm
    release: nfs-provisioner
provisioner: cluster.local/nfs-provisioner-nfs-server-provisioner
reclaimPolicy: Delete

allowVolumeExpansion: true

mountOptions:
  - vers=3
---
# Source: nfs-server-provisioner/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: nfs-provisioner-nfs-server-provisioner
  labels:
    app: nfs-server-provisioner
    chart: nfs-server-provisioner-1.1.3
    heritage: Helm
    release: nfs-provisioner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "endpoints"]
    verbs: ["get"]
  - apiGroups: ["extensions"]
    resources: ["podsecuritypolicies"]
    resourceNames: ["nfs-provisioner"]
    verbs: ["use"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
---
# Source: nfs-server-provisioner/templates/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: nfs-server-provisioner
    chart: nfs-server-provisioner-1.1.3
    heritage: Helm
    release: nfs-provisioner
  name: nfs-provisioner-nfs-server-provisioner
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nfs-provisioner-nfs-server-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner-nfs-server-provisioner
    namespace: default
---
# Source: nfs-server-provisioner/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nfs-provisioner-nfs-server-provisioner
  labels:
    app: nfs-server-provisioner
    chart: nfs-server-provisioner-1.1.3
    heritage: Helm
    release: nfs-provisioner
spec:
  type: ClusterIP
  ports:
    - port: 2049
      targetPort: nfs
      protocol: TCP
      name: nfs
    - port: 2049
      targetPort: nfs-udp
      protocol: UDP
      name: nfs-udp
    - port: 32803
      targetPort: nlockmgr
      protocol: TCP
      name: nlockmgr
    - port: 32803
      targetPort: nlockmgr-udp
      protocol: UDP
      name: nlockmgr-udp
    - port: 20048
      targetPort: mountd
      protocol: TCP
      name: mountd
    - port: 20048
      targetPort: mountd-udp
      protocol: UDP
      name: mountd-udp
    - port: 875
      targetPort: rquotad
      protocol: TCP
      name: rquotad
    - port: 875
      targetPort: rquotad-udp
      protocol: UDP
      name: rquotad-udp
    - port: 111
      targetPort: rpcbind
      protocol: TCP
      name: rpcbind
    - port: 111
      targetPort: rpcbind-udp
      protocol: UDP
      name: rpcbind-udp
    - port: 662
      targetPort: statd
      protocol: TCP
      name: statd
    - port: 662
      targetPort: statd-udp
      protocol: UDP
      name: statd-udp
  selector:
    app: nfs-server-provisioner
    release: nfs-provisioner
---
# Source: nfs-server-provisioner/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nfs-provisioner-nfs-server-provisioner
  labels:
    app: nfs-server-provisioner
    chart: nfs-server-provisioner-1.1.3
    heritage: Helm
    release: nfs-provisioner
spec:
  # TODO: Investigate how/if nfs-provisioner can be scaled out beyond 1 replica
  replicas: 1
  selector:
    matchLabels:
      app: nfs-server-provisioner
      release: nfs-provisioner
  serviceName: nfs-provisioner-nfs-server-provisioner
  template:
    metadata:
      labels:
        app: nfs-server-provisioner
        chart: nfs-server-provisioner-1.1.3
        heritage: Helm
        release: nfs-provisioner
    spec:
      # NOTE: This is 10 seconds longer than the default nfs-provisioner --grace-period value of 90sec
      terminationGracePeriodSeconds: 100
      serviceAccountName: nfs-provisioner-nfs-server-provisioner
      containers:
        - name: nfs-server-provisioner
          image: "quay.io/kubernetes_incubator/nfs-provisioner:v2.3.0"
          imagePullPolicy: IfNotPresent
          ports:
            - name: nfs
              containerPort: 2049
              protocol: TCP
            - name: nfs-udp
              containerPort: 2049
              protocol: UDP
            - name: nlockmgr
              containerPort: 32803
              protocol: TCP
            - name: nlockmgr-udp
              containerPort: 32803
              protocol: UDP
            - name: mountd
              containerPort: 20048
              protocol: TCP
            - name: mountd-udp
              containerPort: 20048
              protocol: UDP
            - name: rquotad
              containerPort: 875
              protocol: TCP
            - name: rquotad-udp
              containerPort: 875
              protocol: UDP
            - name: rpcbind
              containerPort: 111
              protocol: TCP
            - name: rpcbind-udp
              containerPort: 111
              protocol: UDP
            - name: statd
              containerPort: 662
              protocol: TCP
            - name: statd-udp
              containerPort: 662
              protocol: UDP
          securityContext:
            capabilities:
              add:
                - DAC_READ_SEARCH
                - SYS_RESOURCE
          args:
            - "-provisioner=cluster.local/nfs-provisioner-nfs-server-provisioner"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: SERVICE_NAME
              value: nfs-provisioner-nfs-server-provisioner
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
            - name: data
              mountPath: /export
      volumes:
        - name: data
          emptyDir: {}


helm repo add stable https://charts.helm.sh/stable && \
# For deprecation https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner/issues/27

helm install nfs-provisioner stable/nfs-server-provisioner

Anything else we need to know?:

  • Logs form the pod when I run it on kind cluster
I0119 22:08:29.763090       1 main.go:64] Provisioner cluster.local/nfs-provisioner-nfs-server-provisioner specified
I0119 22:08:29.763261       1 main.go:88] Setting up NFS server!
F0119 22:08:33.270755       1 main.go:91] Error setting up NFS server: rpc.statd failed with error: signal: killed, output:
  • The pod runs fine if I use docker run <nfs-server-image>, so to me it seems not a problem with minikube docker, something internal in kind
  • Even runs on k8 with minikube but can’t provision storage

Environment:

  • kind version: (use kind version): kind v0.11.1 go1.16.4 darwin/amd64
  • Kubernetes version: (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:33:37Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"darwin/amd64"}
  • Docker version: (use docker info):
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 15
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: 4144b63817ebcc5b358fc2c8ef95f7cddd709aa7
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.202
 Operating System: Buildroot 2021.02.4
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 5.815GiB
 Name: minikube
 ID: UVRJ:5CMJ:CRH3:5GVD:MEWJ:7ODB:DJYI:OFF3:6EIV:SZBF:HY2N:6QXN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
  provider=hyperkit
 Experimental: false
 Insecure Registries:
  10.96.0.0/12
  192.0.0.0/24
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
  • OS (e.g. from /etc/os-release): Linux minikube 4.19.202 #1 SMP Wed Oct 27 22:52:27 UTC 2021 x86_64 GNU/Linux

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 20 (14 by maintainers)

Most upvoted comments

I opened a special issue, for users wanting to run minikube (kubernetes) on top of minikube (docker):

I’m guessing that more people will get desperate, when they can’t use the “Free” version anymore ?

https://www.docker.com/blog/updating-product-subscriptions/

+2
afbjorklund on Jan 23, 2022

https://github.com/kubernetes-sigs/kind/issues/2597#issuecomment-1019424924 perhaps systemd/libtirpc is older on the hyperkit VM?

+1
BenTheElder on Jan 25, 2022

I think you want to open a new issue for which “docker” to use in kind, it is off-topic here.

+1
afbjorklund on Jan 23, 2022

The minikube OS was supposed to have mount.nfs enabled, and while the kernel is old it is at least 4.19

+1
afbjorklund on Jan 21, 2022

I seem to have looked at this old ugly Fedora installation of a userspace NFS server before…

https://github.com/kubernetes-retired/external-storage/tree/nfs-provisioner-v2.3.0/nfs/deploy/docker/

It was something about not touching with a ten foot pole ?

Hopefully they will make a new release of a ganesha provider.

https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner

Maybe the new fc35 base image works better, than the old fc30 one ?

https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner/blob/master/deploy/base/

Not sure if kind has any supported ReadWriteMany storage provisioner, that works better OOTB

+1
afbjorklund on Jan 20, 2022
Read more comments on GitHub
← kind: HA clusters don't reboot properly
kind: improve error messaging for create cluster" →