kops: 1.17 alpha versions causing regression for kiam?

1. What kops version are you running? The command kops version, will display this information.

Any of the 1.17 alphas so far.

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

seen in 1.17.0-rc.2 through 1.17.3. Works without issue on kops/k8s 1.15 and 1.16 built clusters. ONLY change is bump to 1.17.x.

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

Try to install kiam via it’s included helm chart onto a kops 1.17.x built-cluster

5. What happened after the commands executed?

The kiam-agent daemonsets crashloop

6. What did you expect to happen?

No crashloop.

**7. Please provide your cluster manifest. Execute

Will follow up with this if asked for. Main thing applicable here is we are using CoreDNS

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

From the agent logs w/ gRPC debugging enabled:

kubectl -n kube-system logs kiam-agent-5z8tt
{"level":"info","msg":"started prometheus metric listener 0.0.0.0:9620","time":"2020-02-14T18:13:41Z"}
INFO: 2020/02/14 18:13:41 parsed scheme: "dns"
INFO: 2020/02/14 18:13:46 grpc: failed dns SRV record lookup due to lookup _grpclb._tcp.kiam-server on 100.64.0.10:53: dial udp 100.64.0.10:53: operation was canceled.
WARNING: 2020/02/14 18:13:46 grpc: failed dns A record lookup due to lookup kiam-server on 100.64.0.10:53: dial udp 100.64.0.10:53: operation was canceled.
INFO: 2020/02/14 18:13:46 ccResolverWrapper: got new service config: 
INFO: 2020/02/14 18:13:46 ccResolverWrapper: sending new addresses to cc: []
{"level":"fatal","msg":"error creating server gateway: error dialing grpc server: context deadline exceeded","time":"2020-02-14T18:13:46Z"}

9. Anything else do we need to know?

Bug also posted with kiam folks here: uswitch/kiam#378

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 22 (20 by maintainers)

Most upvoted comments

No release yet. It will go into the next one.