linkerd2: All linkerd components in Init:CrashLoopBackOff

Having below issue with all linkerd components, after circa 48hrs from pod creation.

Environment: kubernetes: GitVersion:“v1.13.2” docker package: docker-1.13.1-103.git7f2769b.el7.centos.x86_64

linkerd version Client version: edge-19.10.1 Server version: edge-19.10.1

Kubernetes running on virtual machines in Google Cloud Platform, with host OS: CentOS Linux release 7.6.1810 (Core)

kubectl get pod -n linkerd
NAME                                      READY   STATUS                  RESTARTS   AGE
linkerd-controller-69d84c4f8c-nd96z       0/2     Init:CrashLoopBackOff   544        2d
linkerd-destination-77bcd7497c-57gqf      0/2     Init:CrashLoopBackOff   544        2d
linkerd-grafana-69b7c55969-mf4h5          0/2     Init:CrashLoopBackOff   544        2d
linkerd-identity-6b6854c8f7-mcw74         0/2     Init:Error              545        2d
linkerd-prometheus-9d59769cc-rjmf8        0/2     Init:CrashLoopBackOff   545        2d
linkerd-proxy-injector-686fd49d85-p2cfc   0/2     Init:CrashLoopBackOff   544        2d
linkerd-sp-validator-77867c74fd-8zgw7     0/2     Init:CrashLoopBackOff   545        2d
linkerd-tap-6c647878c5-bpc2l              0/2     Init:CrashLoopBackOff   545        2d
linkerd-web-7dc9c4b794-vlhqg              0/2     Init:CrashLoopBackOff   544        2d

Snippet from pod description:

kubectl describe pod linkerd-destination-77bcd7497c-57gqf -n linkerd

---
Init Containers:
  linkerd-init:
    Container ID:  docker://e0cd95a592055a5f8e3a758a324a7706a90f74e44d5f753ff697e7a3a379086b
    Image:         gcr.io/linkerd-io/proxy-init:v1.2.0
    Image ID:      docker-pullable://gcr.io/linkerd-io/proxy-init@sha256:c0174438807cdd711867eb1475fba3dd959d764358de4e5f732177e07a75925b
    Port:          <none>
    Host Port:     <none>
    Args:
      --incoming-proxy-port
      4143
      --outgoing-proxy-port
      4140
      --proxy-uid
      2102
      --inbound-ports-to-ignore
      4190,4191
      --outbound-ports-to-ignore
      443
    State:       Waiting
      Reason:    CrashLoopBackOff
    Last State:  Terminated
      Reason:    Error
      Message:   2019/10/11 12:52:18 < iptables: Too many links.

2019/10/11 12:52:18 Will ignore port 4190 on chain PROXY_INIT_REDIRECT
2019/10/11 12:52:18 Will ignore port 4191 on chain PROXY_INIT_REDIRECT
2019/10/11 12:52:18 Will redirect all INPUT ports to proxy
2019/10/11 12:52:18 > iptables -t nat -F PROXY_INIT_OUTPUT
2019/10/11 12:52:18 <
2019/10/11 12:52:18 > iptables -t nat -X PROXY_INIT_OUTPUT
2019/10/11 12:52:18 < iptables: Too many links.

2019/10/11 12:52:18 Ignoring uid 2102
2019/10/11 12:52:18 Will ignore port 443 on chain PROXY_INIT_OUTPUT
2019/10/11 12:52:18 Redirecting all OUTPUT to 4140
2019/10/11 12:52:18 Executing commands:
2019/10/11 12:52:18 > iptables -t nat -N PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-common-chain/1570798338
2019/10/11 12:52:18 < iptables: Chain already exists.

2019/10/11 12:52:18 Aborting firewall configuration
Error: exit status 1
---

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 39 (20 by maintainers)

Most upvoted comments

Thank you guys, think I found problem, that my kube nodes were cleaning unused images every night) But can we fix anyway the problem with cleaning old firewall rules?

0 0 * * * yes | docker system prune -a --volumes

@ptualek sorry that we can’t do anything from a linkerd perspective.

I’m going to close this for now; please reopen or submit a new issue if you want to add anything

@alpeb I’m reasonably sure that the rules I’ve been seeing would result in a broken install. @adleong and I think that erroring out hard is a better solution. That way folks know they’re doing something that they shouldn’t be.