ingress-nginx: MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
NGINX Ingress controller version: 0.45.0 from this config
Kubernetes version (use kubectl version):
% kubectl version --short
Client Version: v1.21.0
Server Version: v1.21.0
% docker -v
Docker version 20.10.5, build 55c4c88
Environment:
-
Cloud provider or hardware configuration:
-
OS (e.g. from /etc/os-release):
Raspbian GNU/Linux 10 (buster) armv7l -
Kernel (e.g.
uname -a):Linux master-pi4 5.10.17-v7l+ #1403 SMP Mon Feb 22 11:33:35 GMT 2021 armv7l GNU/Linux -
Install tools:
-
Others:
What happened:
Ingres controller unable to start
What you expected to happen: Ingres controller starts
it is not the ntftables/ is there maybe a arm image
How to reproduce it:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml
Anything else we need to know:
% kubectl get pod --namespace ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-75557995f8-zf6j7 0/1 ContainerCreating 0 50m
% kubectl describe pods --namespace=ingress-nginx
Name: ingress-nginx-controller-75557995f8-zf6j7
Namespace: ingress-nginx
Priority: 0
Node: node1-pi4/192.168.178.36
Start Time: Mon, 12 Apr 2021 09:27:29 +0200
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=75557995f8
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/ingress-nginx-controller-75557995f8
Containers:
controller:
Container ID:
Image: k8s.gcr.io/ingress-nginx/controller:v0.45.0@sha256:c4390c53f348c3bd4e60a5dd6a11c35799ae78c49388090140b9d72ccede1755
Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--election-id=ingress-controller-leader
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-75557995f8-zf6j7 (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bgcwx (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
kube-api-access-bgcwx:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 62m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-75557995f8-zf6j7 to node1-pi4
Warning FailedMount 42m (x4 over 58m) kubelet Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[kube-api-access-bgcwx webhook-cert]: timed out waiting for the condition
Warning FailedMount 7m42s (x35 over 62m) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
Warning FailedMount 102s (x22 over 60m) kubelet Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert kube-api-access-bgcwx]: timed out waiting for the condition
% kubectl get secret --namespace ingress-nginx
NAME TYPE DATA AGE
default-token-n28m4 kubernetes.io/service-account-token 3 64m
ingress-nginx-admission-token-p6h9j kubernetes.io/service-account-token 3 64m
ingress-nginx-token-4j449 kubernetes.io/service-account-token 3 64m
% kubectl logs -n ingress-nginx ingress-nginx-controller-75557995f8-zf6j7
Error from server (BadRequest): container "controller" in pod "ingress-nginx-controller-75557995f8-zf6j7" is waiting to start: ContainerCreating
% kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default my-nginx-7cf6c4949c-m8j5t 1/1 Running 2 2d2h 10.40.0.0 node1-pi4 <none> <none>
ingress-nginx ingress-nginx-admission-create-x7dpz 1/1 Running 0 4s 10.40.0.2 node1-pi4 <none> <none>
ingress-nginx ingress-nginx-admission-patch-krk89 1/1 Running 0 4s 10.40.0.3 node1-pi4 <none> <none>
ingress-nginx ingress-nginx-controller-75557995f8-5pmqs 0/1 ContainerCreating 0 5s <none> node1-pi4 <none> <none>
kube-system coredns-558bd4d5db-8j2hn 1/1 Running 12 2d9h 10.32.0.2 master-pi4 <none> <none>
kube-system coredns-558bd4d5db-pxx9k 1/1 Running 12 2d9h 10.32.0.3 master-pi4 <none> <none>
kube-system etcd-master-pi4 1/1 Running 19 2d9h 192.168.178.37 master-pi4 <none> <none>
kube-system kube-apiserver-master-pi4 1/1 Running 19 2d9h 192.168.178.37 master-pi4 <none> <none>
kube-system kube-controller-manager-master-pi4 1/1 Running 18 2d9h 192.168.178.37 master-pi4 <none> <none>
kube-system kube-proxy-7jxxl 1/1 Running 3 2d9h 192.168.178.36 node1-pi4 <none> <none>
kube-system kube-proxy-c7htp 1/1 Running 3 2d9h 192.168.178.35 node2-pi3 <none> <none>
kube-system kube-proxy-s92t2 1/1 Running 19 2d9h 192.168.178.37 master-pi4 <none> <none>
kube-system kube-scheduler-master-pi4 1/1 Running 18 2d9h 192.168.178.37 master-pi4 <none> <none>
kube-system weave-net-9gtqz 2/2 Running 28 2d3h 192.168.178.37 master-pi4 <none> <none>
kube-system weave-net-r69z7 2/2 Running 6 2d3h 192.168.178.35 node2-pi3 <none> <none>
kube-system weave-net-xqwz6 2/2 Running 6 2d3h 192.168.178.36 node1-pi4 <none> <none>
Click to expand!
% kubectl logs ingress-nginx-controller-75557995f8-zf6j7 -n ingress-nginx -v10
I0412 15:52:44.741640 18675 loader.go:372] Config loaded from file: /home/pi/.kube/config
I0412 15:52:44.744828 18675 cached_discovery.go:114] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/servergroups.json
I0412 15:52:44.747825 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/apiregistration.k8s.io/v1/serverresources.json
I0412 15:52:44.747882 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/apiregistration.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.748376 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/v1/serverresources.json
I0412 15:52:44.748533 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/flowcontrol.apiserver.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.748765 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/certificates.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.748954 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/apps/v1/serverresources.json
I0412 15:52:44.749040 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/networking.k8s.io/v1/serverresources.json
I0412 15:52:44.749285 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/networking.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.749282 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/apiextensions.k8s.io/v1/serverresources.json
I0412 15:52:44.749886 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/scheduling.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.749942 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/policy/v1beta1/serverresources.json
I0412 15:52:44.749527 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/apiextensions.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.749536 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/extensions/v1beta1/serverresources.json
I0412 15:52:44.749707 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/scheduling.k8s.io/v1/serverresources.json
I0412 15:52:44.749730 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/policy/v1/serverresources.json
I0412 15:52:44.750927 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/storage.k8s.io/v1/serverresources.json
I0412 15:52:44.750093 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/coordination.k8s.io/v1/serverresources.json
I0412 15:52:44.750207 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/rbac.authorization.k8s.io/v1/serverresources.json
I0412 15:52:44.750999 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/discovery.k8s.io/v1/serverresources.json
I0412 15:52:44.750286 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/coordination.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.750555 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/rbac.authorization.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.750595 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/node.k8s.io/v1/serverresources.json
I0412 15:52:44.750822 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/node.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.751255 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/discovery.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.751525 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/events.k8s.io/v1/serverresources.json
I0412 15:52:44.751543 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/authentication.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.751628 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/certificates.k8s.io/v1/serverresources.json
I0412 15:52:44.752015 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/authorization.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.752022 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/autoscaling/v2beta1/serverresources.json
I0412 15:52:44.751296 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/storage.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.751836 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/autoscaling/v1/serverresources.json
I0412 15:52:44.751864 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/authentication.k8s.io/v1/serverresources.json
I0412 15:52:44.752210 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/autoscaling/v2beta2/serverresources.json
I0412 15:52:44.752302 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/authorization.k8s.io/v1/serverresources.json
I0412 15:52:44.751706 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/events.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.752449 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/batch/v1/serverresources.json
I0412 15:52:44.752505 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/batch/v1beta1/serverresources.json
I0412 15:52:44.752658 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/admissionregistration.k8s.io/v1/serverresources.json
I0412 15:52:44.752705 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/admissionregistration.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.754417 18675 cached_discovery.go:114] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/servergroups.json
I0412 15:52:44.755072 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/events.k8s.io/v1/serverresources.json
I0412 15:52:44.755076 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/apiregistration.k8s.io/v1/serverresources.json
I0412 15:52:44.755087 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/flowcontrol.apiserver.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.755287 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/apiregistration.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.755667 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/apps/v1/serverresources.json
I0412 15:52:44.755743 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/v1/serverresources.json
I0412 15:52:44.755867 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/policy/v1beta1/serverresources.json
I0412 15:52:44.755910 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/apiextensions.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.756101 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/events.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.756114 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/authentication.k8s.io/v1/serverresources.json
I0412 15:52:44.756116 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/rbac.authorization.k8s.io/v1/serverresources.json
I0412 15:52:44.756234 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/authentication.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.756303 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/rbac.authorization.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.756379 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/authorization.k8s.io/v1/serverresources.json
I0412 15:52:44.756489 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/storage.k8s.io/v1/serverresources.json
I0412 15:52:44.756530 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/authorization.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.756667 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/autoscaling/v1/serverresources.json
I0412 15:52:44.756802 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/autoscaling/v2beta1/serverresources.json
I0412 15:52:44.756840 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/storage.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.756935 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/autoscaling/v2beta2/serverresources.json
I0412 15:52:44.756986 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/admissionregistration.k8s.io/v1/serverresources.json
I0412 15:52:44.758087 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/networking.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.757094 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/batch/v1/serverresources.json
I0412 15:52:44.757134 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/admissionregistration.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.758144 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/node.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.757232 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/batch/v1beta1/serverresources.json
I0412 15:52:44.757280 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/apiextensions.k8s.io/v1/serverresources.json
I0412 15:52:44.757384 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/certificates.k8s.io/v1/serverresources.json
I0412 15:52:44.757402 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/coordination.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.757521 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/scheduling.k8s.io/v1/serverresources.json
I0412 15:52:44.757549 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/certificates.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.757644 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/scheduling.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.757763 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/coordination.k8s.io/v1/serverresources.json
I0412 15:52:44.757769 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/networking.k8s.io/v1/serverresources.json
I0412 15:52:44.757905 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/extensions/v1beta1/serverresources.json
I0412 15:52:44.758017 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/policy/v1/serverresources.json
I0412 15:52:44.758279 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/node.k8s.io/v1/serverresources.json
I0412 15:52:44.758401 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/discovery.k8s.io/v1beta1/serverresources.json
I0412 15:52:44.758635 18675 cached_discovery.go:71] returning cached discovery info from /home/pi/.kube/cache/discovery/192.168.178.37_6443/discovery.k8s.io/v1/serverresources.json
I0412 15:52:44.761607 18675 round_trippers.go:435] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.21.0 (linux/arm) kubernetes/cb303e6" 'https://192.168.178.37:6443/api/v1/namespaces/ingress-nginx/pods/ingress-nginx-controller-75557995f8-zf6j7'
I0412 15:52:44.838289 18675 round_trippers.go:454] GET https://192.168.178.37:6443/api/v1/namespaces/ingress-nginx/pods/ingress-nginx-controller-75557995f8-zf6j7 200 OK in 76 milliseconds
I0412 15:52:44.838370 18675 round_trippers.go:460] Response Headers:
I0412 15:52:44.838418 18675 round_trippers.go:463] Cache-Control: no-cache, private
I0412 15:52:44.838444 18675 round_trippers.go:463] Content-Type: application/json
I0412 15:52:44.838468 18675 round_trippers.go:463] X-Kubernetes-Pf-Flowschema-Uid: d0d1156f-52a3-4392-be05-50089edd99cd
I0412 15:52:44.838509 18675 round_trippers.go:463] X-Kubernetes-Pf-Prioritylevel-Uid: 25ad5dcf-f661-47a4-a95b-2087ec4301fb
I0412 15:52:44.838533 18675 round_trippers.go:463] Date: Mon, 12 Apr 2021 13:52:44 GMT
I0412 15:52:44.839604 18675 request.go:1123] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"ingress-nginx-controller-75557995f8-zf6j7","generateName":"ingress-nginx-controller-75557995f8-","namespace":"ingress-nginx","uid":"4e5e42c2-6ec3-4780-9c1c-d7146fd934e0","resourceVersion":"229422","creationTimestamp":"2021-04-12T07:27:29Z","labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","pod-template-hash":"75557995f8"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"ingress-nginx-controller-75557995f8","uid":"0685fe96-cfeb-45b6-89fd-c2c6ab1f020e","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-04-12T07:27:29Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/name":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"0685fe96-cfeb-45b6-89fd-c2c6ab1f020e\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"controller\"}":{".":{},"f:args":{},"f:env":{".":{},"k:{\"name\":\"LD_PRELOAD\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"POD_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:lifecycle":{".":{},"f:preStop":{".":{},"f:exec":{".":{},"f:command":{}}}},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":80,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":443,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":8443,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{},"f:capabilities":{".":{},"f:add":{},"f:drop":{}},"f:runAsUser":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/usr/local/certificates/\"}":{".":{},"f:mountPath":{},"f:name":{},"f:readOnly":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:nodeSelector":{".":{},"f:kubernetes.io/os":{}},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"webhook-cert\"}":{".":{},"f:name":{},"f:secret":{".":{},"f:defaultMode":{},"f:secretName":{}}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-12T07:27:29Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"webhook-cert","secret":{"secretName":"ingress-nginx-admission","defaultMode":420}},{"name":"kube-api-access-bgcwx","projected":{"sources":[{"serviceAccountToken":{"expirationSeconds":3607,"path":"token"}},{"configMap":{"name":"kube-root-ca.crt","items":[{"key":"ca.crt","path":"ca.crt"}]}},{"downwardAPI":{"items":[{"path":"namespace","fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}]}}],"defaultMode":420}}],"containers":[{"name":"controller","image":"k8s.gcr.io/ingress-nginx/controller:v0.45.0@sha256:c4390c53f348c3bd4e60a5dd6a11c35799ae78c49388090140b9d72ccede1755","args":["/nginx-ingress-controller","--election-id=ingress-controller-leader","--ingress-class=nginx","--configmap=$(POD_NAMESPACE)/ingress-nginx-controller","--validating-webhook=:8443","--validating-webhook-certificate=/usr/local/certificates/cert","--validating-webhook-key=/usr/local/certificates/key"],"ports":[{"name":"http","containerPort":80,"protocol":"TCP"},{"name":"https","containerPort":443,"protocol":"TCP"},{"name":"webhook","containerPort":8443,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"LD_PRELOAD","value":"/usr/local/lib/libmimalloc.so"}],"resources":{"requests":{"cpu":"100m","memory":"90Mi"}},"volumeMounts":[{"name":"webhook-cert","readOnly":true,"mountPath":"/usr/local/certificates/"},{"name":"kube-api-access-bgcwx","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"initialDelaySeconds":10,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":5},"readinessProbe":{"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"initialDelaySeconds":10,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"lifecycle":{"preStop":{"exec":{"command":["/wait-shutdown"]}}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["ALL"]},"runAsUser":101,"allowPrivilegeEscalation":true}}],"restartPolicy":"Always","terminationGracePeriodSeconds":300,"dnsPolicy":"ClusterFirst","nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"ingress-nginx","serviceAccount":"ingress-nginx","nodeName":"node1-pi4","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-12T07:27:29Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-12T07:27:29Z","reason":"ContainersNotReady","message":"containers with unready status: [controller]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-12T07:27:29Z","reason":"ContainersNotReady","message":"containers with unready status: [controller]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-12T07:27:29Z"}],"hostIP":"192.168.178.36","startTime":"2021-04-12T07:27:29Z","containerStatuses":[{"name":"controller","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"k8s.gcr.io/ingress-nginx/controller:v0.45.0@sha256:c4390c53f348c3bd4e60a5dd6a11c35799ae78c49388090140b9d72ccede1755","imageID":"","started":false}],"qosClass":"Burstable"}}
I0412 15:52:44.862320 18675 round_trippers.go:435] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.21.0 (linux/arm) kubernetes/cb303e6" 'https://192.168.178.37:6443/api/v1/namespaces/ingress-nginx/pods/ingress-nginx-controller-75557995f8-zf6j7/log'
I0412 15:52:44.930844 18675 round_trippers.go:454] GET https://192.168.178.37:6443/api/v1/namespaces/ingress-nginx/pods/ingress-nginx-controller-75557995f8-zf6j7/log 400 Bad Request in 68 milliseconds
I0412 15:52:44.930918 18675 round_trippers.go:460] Response Headers:
I0412 15:52:44.931246 18675 round_trippers.go:463] Cache-Control: no-cache, private
I0412 15:52:44.931283 18675 round_trippers.go:463] Content-Type: application/json
I0412 15:52:44.931329 18675 round_trippers.go:463] Content-Length: 231
I0412 15:52:44.931354 18675 round_trippers.go:463] Date: Mon, 12 Apr 2021 13:52:44 GMT
I0412 15:52:44.931577 18675 request.go:1123] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"controller\" in pod \"ingress-nginx-controller-75557995f8-zf6j7\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400}
I0412 15:52:44.933258 18675 helpers.go:216] server response object: [{
"metadata": {},
"status": "Failure",
"message": "container \"controller\" in pod \"ingress-nginx-controller-75557995f8-zf6j7\" is waiting to start: ContainerCreating",
"reason": "BadRequest",
"code": 400
}]
F0412 15:52:44.933697 18675 helpers.go:115] Error from server (BadRequest): container "controller" in pod "ingress-nginx-controller-75557995f8-zf6j7" is waiting to start: ContainerCreating
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0x2574301, 0x0, 0xbf, 0x1e6)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1021 +0x94
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x2563080, 0x3, 0x0, 0x0, 0x2cc0ae0, 0x1cc8f67, 0xa, 0x73, 0xfda800)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:970 +0x110
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x2563080, 0x3, 0x0, 0x0, 0x0, 0x0, 0x2, 0x3103750, 0x1, 0x1)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:733 +0x120
k8s.io/kubernetes/vendor/k8s.io/klog/v2.FatalDepth(...)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1495
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.fatal(0x2e53320, 0x90, 0x1)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:93 +0x1dc
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.checkErr(0x1797188, 0x3261cc0, 0x1633168)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:188 +0x6e0
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:115
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/logs.NewCmdLogs.func1(0x30374a0, 0x3027620, 0x1, 0x4)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/logs/logs.go:158 +0xf0
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0x30374a0, 0x3027600, 0x4, 0x4, 0x30374a0, 0x3027600)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:854 +0x1f4
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x2d491e0, 0x2d08060, 0x2d08030, 0x6)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958 +0x274
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
main.main()
_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:49 +0x1e8
goroutine 18 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x2563080)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1164 +0x70
created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:418 +0x100
goroutine 6 [select]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x16330f8, 0x1797368, 0x2ca4000, 0x1, 0x2efd4c0)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0xf0
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x16330f8, 0x2a05f200, 0x1, 0x0, 0x0, 0x1, 0x2efd4c0)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x80
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0x16330f8, 0x2a05f200, 0x1, 0x2efd4c0)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x48
created by k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs.InitLogs
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs/logs.go:51 +0x80
goroutine 98 [IO wait]:
internal/poll.runtime_pollWait(0xa6c98068, 0x72, 0xffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x44
internal/poll.(*pollDesc).wait(0x3313194, 0x72, 0x1d00, 0x1d7c, 0xffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x30
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0x3313180, 0x32d0000, 0x1d7c, 0x1d7c, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1a4
net.(*netFD).Read(0x3313180, 0x32d0000, 0x1d7c, 0x1d7c, 0x32d0005, 0x1633b70, 0x1d77)
/usr/local/go/src/net/fd_posix.go:55 +0x38
net.(*conn).Read(0x2c98760, 0x32d0000, 0x1d7c, 0x1d7c, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x64
crypto/tls.(*atLeastReader).Read(0x30122f0, 0x32d0000, 0x1d7c, 0x1d7c, 0x10, 0x30122f0, 0x2c566c0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x78
bytes.(*Buffer).ReadFrom(0x306e96c, 0x1796544, 0x30122f0, 0x106, 0x205174, 0x1361848, 0x14dee78)
/usr/local/go/src/bytes/buffer.go:204 +0xa4
crypto/tls.(*Conn).readFromUntil(0x306e800, 0x17981f0, 0x2c98760, 0x5, 0x2c98760, 0xf0)
/usr/local/go/src/crypto/tls/conn.go:798 +0xc4
crypto/tls.(*Conn).readRecordOrCCS(0x306e800, 0x0, 0x0, 0x30122a0)
/usr/local/go/src/crypto/tls/conn.go:605 +0xfc
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0x306e800, 0x2e57000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x154
bufio.(*Reader).Read(0x2c37170, 0x338c9a0, 0x9, 0x9, 0x0, 0x0, 0x1)
/usr/local/go/src/bufio/bufio.go:227 +0x22c
io.ReadAtLeast(0x1796418, 0x2c37170, 0x338c9a0, 0x9, 0x9, 0x9, 0x1633900, 0x4a09c8, 0x1633901)
/usr/local/go/src/io/io.go:328 +0x6c
io.ReadFull(...)
/usr/local/go/src/io/io.go:347
k8s.io/kubernetes/vendor/golang.org/x/net/http2.readFrameHeader(0x338c9a0, 0x9, 0x9, 0x1796418, 0x2c37170, 0x0, 0x0, 0x0, 0x32c7fd4, 0x331f340)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:237 +0x54
k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0x338c980, 0x331c8e8, 0x0, 0x0, 0x0)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:492 +0x74
k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run(0x32c7fd4, 0x0, 0x0)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:1819 +0xdc
k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0x2c380f0)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:1741 +0x60
created by k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*Transport).newClientConn
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:705 +0x580
/kind bug
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 62 (37 by maintainers)
@sumit-badaya, you are not helping.
The above commands are one way to put the controller in a namespace of your choice.
Any normal k8s admin woiuld be aware of this.
But you are neither providing steps for a developer to reproduce the problem, nor are you helping anyone investigate the problem on their own with some specifics. Can you move your comments to a issue dedicated to your problem and please not hijack this
skepticism, personal comments, trash-talk does not help developers or the person who posted that ARM issue. Like I said, the reason for not posting the file here is to avoid adding unrelated artifact in this case. Don’t show your unhelpful side and please open another namespace issue and I will attach the file there and then tell me if it works or not. So sad to see filling unrelated content here. You don’t manage issues so you can’t feel other people’s pain.
That is not documented procedure. Your options are to study every single object involved and do it yourself or create a new issue asking the project developers for additional docs on that. In either case it is unrelated to this arm issue.
Thanks, ; Long
On Sun, 18 Apr, 2021, 9:40 PM Robyn Ward, @.***> wrote:
Then you gotta delete the comments that has me mentioned in it. Please do the needful so i can delete mine.
Because that makes sense coz you will have me mentioned while my comments wont be there at all in entire thread. Please delete all that contains my username mentioned.
On Sun, 18 Apr, 2021, 11:30 pm Long, @.***> wrote:
@LeducH did you get this resolve? Please help others if you did.
create the secret manually first and then comment