jaeger-operator: DaemonSet test failing

Currently, the DaemonSet e2e test is failing:

Running end-to-end tests...
time="2018-11-20T13:25:34+01:00" level=info msg="passing &{{Jaeger io.jaegertracing/v1alpha1} {agent-as-daemonset  jaeger-jaeger-group-daemonset-1542716724    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {allInOne { {map[]} {[] [] map[] {map[] map[]}}} {0  {map[]} {[] [] map[] {map[] map[]}}} {0  {map[]} {[] [] map[] {map[] map[]}}} {DaemonSet  {map[log-level:debug]} {[] [] map[] {map[] map[]}}} { {map[]} {<nil>   }} {<nil>  {[] [] map[] {map[] map[]}}} {[] [] map[] {map[] map[]}}} {}}"
time="2018-11-20T13:27:05+01:00" level=info msg="passing &{{Jaeger io.jaegertracing/v1alpha1} {with-cassandra  jaeger-jaeger-group-cassandra-1542716820    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {allInOne { {map[]} {[] [] map[] {map[] map[]}}} {0  {map[]} {[] [] map[] {map[] map[]}}} {0  {map[]} {[] [] map[] {map[] map[]}}} {  {map[]} {[] [] map[] {map[] map[]}}} {cassandra {map[cassandra.servers:cassandra.default.svc]} {<nil>   }} {<nil>  {[] [] map[] {map[] map[]}}} {[] [] map[] {map[] map[]}}} {}}"
time="2018-11-20T13:27:30+01:00" level=info msg="passing &{{Jaeger io.jaegertracing/v1alpha1} {my-jaeger  jaeger-jaeger-group-my-other-jaeger-1542716840    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {allInOne { {map[log-level:debug memory.max-traces:10000]} {[] [] map[] {map[] map[]}}} {0  {map[]} {[] [] map[] {map[] map[]}}} {0  {map[]} {[] [] map[] {map[] map[]}}} {  {map[]} {[] [] map[] {map[] map[]}}} { {map[]} {<nil>   }} {<nil>  {[] [] map[] {map[] map[]}}} {[] [] map[] {map[] map[]}}} {}}"
time="2018-11-20T13:27:30+01:00" level=info msg="passing &{{Jaeger io.jaegertracing/v1alpha1} {my-jaeger  jaeger-jaeger-group-my-jaeger-1542716840    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {allInOne { {map[memory.max-traces:10000 log-level:debug]} {[] [] map[] {map[] map[]}}} {0  {map[]} {[] [] map[] {map[] map[]}}} {0  {map[]} {[] [] map[] {map[] map[]}}} {  {map[]} {[] [] map[] {map[] map[]}}} { {map[]} {<nil>   }} {<nil>  {[] [] map[] {map[] map[]}}} {[] [] map[] {map[] map[]}}} {}}"
--- FAIL: TestJaeger (151.38s)
    --- FAIL: TestJaeger/jaeger-group (115.68s)
        --- FAIL: TestJaeger/jaeger-group/daemonset (70.16s)
            client.go:57: resource type Role with namespace/name (jaeger-jaeger-group-daemonset-1542716724/jaeger-operator) created
            client.go:57: resource type RoleBinding with namespace/name (jaeger-jaeger-group-daemonset-1542716724/default-account-jaeger-operator) created
            client.go:57: resource type Deployment with namespace/name (jaeger-jaeger-group-daemonset-1542716724/jaeger-operator) created
            jaeger_test.go:50: Initialized cluster resources
            wait_util.go:45: Waiting for full availability of jaeger-operator deployment (0/1)
            wait_util.go:51: Deployment available (1/1)
            client.go:57: resource type Jaeger with namespace/name (jaeger-jaeger-group-daemonset-1542716724/agent-as-daemonset) created
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            wait_util.go:55: Waiting for full availability of agent-as-daemonset-agent-daemonset daemonsets (0/1)
            daemonset.go:29: timed out waiting for the condition
            client.go:75: resource type Jaeger with namespace/name (jaeger-jaeger-group-daemonset-1542716724/agent-as-daemonset) successfully deleted
            client.go:75: resource type Deployment with namespace/name (jaeger-jaeger-group-daemonset-1542716724/jaeger-operator) successfully deleted
            client.go:75: resource type RoleBinding with namespace/name (jaeger-jaeger-group-daemonset-1542716724/default-account-jaeger-operator) successfully deleted
            client.go:75: resource type Role with namespace/name (jaeger-jaeger-group-daemonset-1542716724/jaeger-operator) successfully deleted
FAIL
FAIL	github.com/jaegertracing/jaeger-operator/test/e2e	151.460s

According to git bisect:

13d7cc5db1b3b9effbd262e9f0e77d8a1b76d139 is the first bad commit
commit 13d7cc5db1b3b9effbd262e9f0e77d8a1b76d139
Author: Juraci Paixão Kröhling <juraci.github@kroehling.de>
Date:   Tue Nov 13 16:29:46 2018 +0100

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 17 (11 by maintainers)

Most upvoted comments

Not yet I am waiting for https://github.com/jaegertracing/jaeger-ui/pull/263

pavolloffay on Nov 21, 2018

I’m noticing the same thing happening when trying to run the agent as a DaemonSet, the agent-daemonset never becomes available. i.e.

jaeger-agent-daemonset-vbnzv        0/1       Running   0          7m
jaeger-collector-745dddf5c6-thxxx   1/1       Running   0          7m
jaeger-operator-7f5d55ffb6-qdzt2    1/1       Running   0          8m
jaeger-query-74bb5dc84d-6ppqs       1/1       Running   0          7m

The logs I get from this Pod are:

{"level":"warn","ts":1542730394.6390572,"caller":"tchannel/flags.go:67","msg":"Using deprecated configuration","option":"collector.host-port"} 
{"level":"info","ts":1542730394.639524,"caller":"tchannel/builder.go:94","msg":"Enabling service discovery","service":"jaeger-collector"} 
{"level":"info","ts":1542730394.6397831,"caller":"peerlistmgr/peer_list_mgr.go:111","msg":"Registering active peer","peer":"jaeger-collector:14267"} 
{"level":"info","ts":1542730394.6407661,"caller":"agent/main.go:75","msg":"Starting agent"} 
{"level":"info","ts":1542730395.6402557,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1} 
{"level":"info","ts":1542730395.640376,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"jaeger-collector:14267"} 
{"level":"error","ts":1542730395.6445525,"caller":"peerlistmgr/peer_list_mgr.go:171","msg":"Unable to connect","host:port":"jaeger-collector:14267","connCheckTimeout":0.25,"error":"dial tcp: lookup jaeger-collector on 10.96.0.10:53: server misbehaving","stacktrace":"github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).ensureConnectionsnt/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:171ngithub.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).maintainConnectionsnt/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:101"} 
{"level":"info","ts":1542730396.6405394,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1} 
{"level":"info","ts":1542730396.6407301,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"jaeger-collector:14267"} 
{"level":"error","ts":1542730396.651564,"caller":"peerlistmgr/peer_list_mgr.go:171","msg":"Unable to connect","host:port":"jaeger-collector:14267","connCheckTimeout":0.25,"error":"dial tcp: lookup jaeger-collector on 10.96.0.10:53: server misbehaving","stacktrace":"github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).ensureConnectionsnt/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:171ngithub.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).maintainConnectionsnt/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:101"} 
{"level":"info","ts":1542730397.6404357,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1} 
{"level":"info","ts":1542730397.6408088,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"jaeger-collector:14267"} 
{"level":"error","ts":1542730397.6513267,"caller":"peerlistmgr/peer_list_mgr.go:171","msg":"Unable to connect","host:port":"jaeger-collector:14267","connCheckTimeout":0.25,"error":"dial tcp: lookup jaeger-collector on 10.96.0.10:53: server misbehaving","stacktrace":"github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).ensureConnectionsnt/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:171ngithub.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).maintainConnectionsnt/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:101"} 
{"level":"info","ts":1542730398.6404321,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1} 
{"level":"info","ts":1542730398.6406636,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"jaeger-collector:14267"} 
{"level":"error","ts":1542730398.6499462,"caller":"peerlistmgr/peer_list_mgr.go:171","msg":"Unable to connect","host:port":"jaeger-collector:14267","connCheckTimeout":0.25,"error":"dial tcp: lookup jaeger-collector on 10.96.0.10:53: server misbehaving","stacktrace":"github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).ensureConnectionsnt/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:171ngithub.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).maintainConnectionsnt/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:101"} 
{"level":"info","ts":1542730399.640582,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1} 
{"level":"info","ts":1542730399.640759,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"jaeger-collector:14267"} 
{"level":"error","ts":1542730399.6502535,"caller":"peerlistmgr/peer_list_mgr.go:171","msg":"Unable to connect","host:port":"jaeger-collector:14267","connCheckTimeout":0.25,"error":"dial tcp: lookup jaeger-collector on 10.96.0.10:53: server misbehaving","stacktrace":"github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).ensureConnectionsnt/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:171ngithub.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).maintainConnectionsnt/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:101"} 
{"level":"info","ts":1542730400.640373,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1} 
{"level":"info","ts":1542730400.6405044,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"jaeger-collector:14267"} 
{"level":"info","ts":1542730400.6489508,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"[::]:14267"}

Seems like it connects to the collector and then ‘stops’.

The readiness probe is the thing that is stopping the agent becoming available,

  Warning  Unhealthy              4m (x31 over 9m)  kubelet, minikube  Readiness probe failed: HTTP probe failed with statuscode: 400

dlmiddlecote on Nov 20, 2018