cilium: CI: endpoint in not-ready state due to "clang-3.8: error: unable to execute command: Killed"
Output:
/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:376
Endpoints are not ready after timeout
Expected
<*errors.errorString | 0xc420377760>: {
s: "Timeout reached: cannot retrieve endpoints",
}
to be nil
/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/k8sT/Updates.go:275
Test stdout:
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
⚠️ Number of "level=warning" in logs: 41
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Cilium pods: [cilium-9lrvn cilium-vr9kx]
Netpols loaded:
CiliumNetworkPolicies loaded: default::l7-policy
Endpoint Policy Enforcement:
app1-7c5785f7f7-mpshw => ingress
app2-69f6b59665-ncnhn => none
app3-bd87968cd-zcxwf => none
coredns-687db6485c-gvh8z => none
prometheus-core-6546477dc8-bnpfn => none
app1-7c5785f7f7-9wjm4 => ingress
Cilium agent "cilium-9lrvn": Status: Ok Health: Ok Nodes "k8s2 k8s1" ContinerRuntime: Ok Kubernetes: Ok KVstore: Ok Controllers: Total 24 Failed 0
Cilium agent "cilium-vr9kx": Status: Ok Health: Ok Nodes "k8s1 k8s2" ContinerRuntime: Ok Kubernetes: Ok KVstore: Ok Controllers: Total 34 Failed 0
Test stderr:
STEP: Cilium "v1.2" is installed and running
STEP: Installing Microscope
STEP: Installing kube-dns
STEP: Creating some endpoints and L7 policy
STEP: Making L7 requests between endpoints
STEP: Updating cilium to master image
STEP: Checking that installed image is "k8s1:5000/cilium/cilium-dev:latest"
STEP: Making L7 requests between endpoints
STEP: Downgrading cilium to v1.2 image
STEP: Checking that installed image is "docker.io/cilium/cilium:v1.2"
=== Test Finished at 2018-09-25T14:52:19Z====
===================== TEST FAILED =====================
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
default app1-7c5785f7f7-9wjm4 1/1 Running 0 6m 10.10.0.61 k8s1 <none>
default app1-7c5785f7f7-mpshw 1/1 Running 0 6m 10.10.0.28 k8s1 <none>
default app2-69f6b59665-ncnhn 1/1 Running 0 6m 10.10.0.51 k8s1 <none>
default app3-bd87968cd-zcxwf 1/1 Running 0 6m 10.10.0.83 k8s1 <none>
kube-system cilium-9lrvn 1/1 Running 0 5m 192.168.36.12 k8s2 <none>
kube-system cilium-vr9kx 1/1 Running 0 5m 192.168.36.11 k8s1 <none>
kube-system coredns-687db6485c-gvh8z 1/1 Running 0 6m 10.10.1.46 k8s2 <none>
kube-system etcd-k8s1 1/1 Running 0 40m 192.168.36.11 k8s1 <none>
kube-system kube-apiserver-k8s1 1/1 Running 0 40m 192.168.36.11 k8s1 <none>
kube-system kube-controller-manager-k8s1 1/1 Running 0 40m 192.168.36.11 k8s1 <none>
kube-system kube-proxy-jqnsc 1/1 Running 0 41m 192.168.36.11 k8s1 <none>
kube-system kube-proxy-t2xgb 1/1 Running 0 34m 192.168.36.12 k8s2 <none>
kube-system kube-scheduler-k8s1 1/1 Running 0 40m 192.168.36.11 k8s1 <none>
monitoring prometheus-core-6546477dc8-bnpfn 1/1 Running 0 34m 10.10.0.124 k8s1 <none>
Stderr:
cmd: kubectl exec -n kube-system cilium-9lrvn -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
4198 Disabled Disabled 4 reserved:health f00d::a0a:100:0:1066 10.10.1.4 ready
7733 Disabled Disabled 60379 k8s:io.cilium.k8s.policy.serviceaccount=coredns f00d::a0a:100:0:1e35 10.10.1.46 ready
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
Stderr:
cmd: kubectl exec -n kube-system cilium-vr9kx -- cilium endpoint list
Exitcode: 0
Stdout:
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
3264 Enabled Disabled 15133 k8s:id=app1 f00d::a0a:0:0:cc0 10.10.0.28 ready
k8s:io.cilium.k8s.policy.serviceaccount=app1-account
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testapp
32714 Enabled Disabled 15133 k8s:id=app1 f00d::a0a:0:0:7fca 10.10.0.61 ready
k8s:io.cilium.k8s.policy.serviceaccount=app1-account
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testapp
49277 Disabled Disabled 32890 k8s:id=app3 f00d::a0a:0:0:c07d 10.10.0.83 ready
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testapp
54713 Disabled Disabled 51325 k8s:appSecond=true f00d::a0a:0:0:d5b9 10.10.0.51 ready
k8s:id=app2
k8s:io.cilium.k8s.policy.serviceaccount=app2-account
k8s:io.kubernetes.pod.namespace=default
k8s:zgroup=testapp
58302 Disabled Disabled 4 reserved:health f00d::a0a:0:0:e3be 10.10.0.25 ready
Stderr:
Console output from Jenkins including timestamps:
14:44:50 K8sUpdates
14:44:50 Tests upgrade and downgrade from a Cilium stable image to master
14:44:50 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:376
14:45:38 STEP: Cilium "v1.2" is installed and running
14:45:38 STEP: Installing Microscope
14:45:49 STEP: Installing kube-dns
14:45:49 STEP: Creating some endpoints and L7 policy
14:46:05 STEP: Making L7 requests between endpoints
14:46:05 STEP: Updating cilium to master image
14:46:28 STEP: Checking that installed image is "k8s1:5000/cilium/cilium-dev:latest"
14:46:29 STEP: Making L7 requests between endpoints
14:46:29 STEP: Downgrading cilium to v1.2 image
14:46:48 STEP: Checking that installed image is "docker.io/cilium/cilium:v1.2"
14:52:28 === Test Finished at 2018-09-25T14:52:19Z====
Logs: d474ea02_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 25 (25 by maintainers)
@aanm just enjoys blaming my PRs 😄