linkerd2: All linkerd components in Init:CrashLoopBackOff
Having below issue with all linkerd components, after circa 48hrs from pod creation.
Environment: kubernetes: GitVersion:“v1.13.2” docker package: docker-1.13.1-103.git7f2769b.el7.centos.x86_64
linkerd version Client version: edge-19.10.1 Server version: edge-19.10.1
Kubernetes running on virtual machines in Google Cloud Platform, with host OS: CentOS Linux release 7.6.1810 (Core)
kubectl get pod -n linkerd
NAME READY STATUS RESTARTS AGE
linkerd-controller-69d84c4f8c-nd96z 0/2 Init:CrashLoopBackOff 544 2d
linkerd-destination-77bcd7497c-57gqf 0/2 Init:CrashLoopBackOff 544 2d
linkerd-grafana-69b7c55969-mf4h5 0/2 Init:CrashLoopBackOff 544 2d
linkerd-identity-6b6854c8f7-mcw74 0/2 Init:Error 545 2d
linkerd-prometheus-9d59769cc-rjmf8 0/2 Init:CrashLoopBackOff 545 2d
linkerd-proxy-injector-686fd49d85-p2cfc 0/2 Init:CrashLoopBackOff 544 2d
linkerd-sp-validator-77867c74fd-8zgw7 0/2 Init:CrashLoopBackOff 545 2d
linkerd-tap-6c647878c5-bpc2l 0/2 Init:CrashLoopBackOff 545 2d
linkerd-web-7dc9c4b794-vlhqg 0/2 Init:CrashLoopBackOff 544 2d
Snippet from pod description:
kubectl describe pod linkerd-destination-77bcd7497c-57gqf -n linkerd
---
Init Containers:
linkerd-init:
Container ID: docker://e0cd95a592055a5f8e3a758a324a7706a90f74e44d5f753ff697e7a3a379086b
Image: gcr.io/linkerd-io/proxy-init:v1.2.0
Image ID: docker-pullable://gcr.io/linkerd-io/proxy-init@sha256:c0174438807cdd711867eb1475fba3dd959d764358de4e5f732177e07a75925b
Port: <none>
Host Port: <none>
Args:
--incoming-proxy-port
4143
--outgoing-proxy-port
4140
--proxy-uid
2102
--inbound-ports-to-ignore
4190,4191
--outbound-ports-to-ignore
443
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Message: 2019/10/11 12:52:18 < iptables: Too many links.
2019/10/11 12:52:18 Will ignore port 4190 on chain PROXY_INIT_REDIRECT
2019/10/11 12:52:18 Will ignore port 4191 on chain PROXY_INIT_REDIRECT
2019/10/11 12:52:18 Will redirect all INPUT ports to proxy
2019/10/11 12:52:18 > iptables -t nat -F PROXY_INIT_OUTPUT
2019/10/11 12:52:18 <
2019/10/11 12:52:18 > iptables -t nat -X PROXY_INIT_OUTPUT
2019/10/11 12:52:18 < iptables: Too many links.
2019/10/11 12:52:18 Ignoring uid 2102
2019/10/11 12:52:18 Will ignore port 443 on chain PROXY_INIT_OUTPUT
2019/10/11 12:52:18 Redirecting all OUTPUT to 4140
2019/10/11 12:52:18 Executing commands:
2019/10/11 12:52:18 > iptables -t nat -N PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-common-chain/1570798338
2019/10/11 12:52:18 < iptables: Chain already exists.
2019/10/11 12:52:18 Aborting firewall configuration
Error: exit status 1
---
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 39 (20 by maintainers)
Thank you guys, think I found problem, that my kube nodes were cleaning unused images every night) But can we fix anyway the problem with cleaning old firewall rules?
@ptualek sorry that we can’t do anything from a linkerd perspective.
I’m going to close this for now; please reopen or submit a new issue if you want to add anything
@alpeb I’m reasonably sure that the rules I’ve been seeing would result in a broken install. @adleong and I think that erroring out hard is a better solution. That way folks know they’re doing something that they shouldn’t be.