test-infra: [SQ-blocking] DumpClusterLogs failing in federation presubmit

For e.g - https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/50310/pull-kubernetes-federation-e2e-gce/19456/build-log.txt

Failing due to the following reason:

I0808 15:18:39.593] Master not detected. Is the cluster up?
I0808 15:18:39.593] Dumping logs from nodes locally to '/workspace/_artifacts'
I0808 15:18:39.594] Detecting nodes in the cluster
W0808 15:18:40.350] ./cluster/log-dump/log-dump.sh: line 223: NODE_NAMES[@]: unbound variable

Seems like a bug which has gotten exposed by the recent change https://github.com/kubernetes/test-infra/pull/3974 which makes DumpClusterLogs happen only inside kubetest instead of also doing it in ginkgo.

cc @fejta @krzyzacy

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 24 (24 by maintainers)

Commits related to this issue

Merge pull request #50333 from shyamjvs/add-debug-to-logexporter Automatic merge from submit-queue (batch tested with PRs 50173, 50324, 50288, 50263, 50333) Add debug logs to log-dump ref https://g... — committed to kubernetes/kubernetes by deleted user 7 years ago

Most upvoted comments

So after I added extra logging it’s clear why it’s failing (ref: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/5950/build-log.txt) -

W0809 14:32:50.556] Trying to find master named 'bootstrap-e2e-master'
W0809 14:32:50.556] Looking for address 'bootstrap-e2e-master-ip'
W0809 14:32:51.190] ERROR: (gcloud.compute.addresses.describe) Could not fetch resource:
W0809 14:32:51.190]  - The resource 'projects/k8s-jkns-e2e-gce-federation/regions/us-central1/addresses/bootstrap-e2e-master-ip' was not found

It is picking up the wrong master name, which is calculated as ‘${INSTANCE_PREFIX}-master’ and INSTANCE_PREFIX=bootstrap-e2e. The the real master names seem to be ‘prow-us-central1-a-master’ and ‘prow-us-central1-f-master’.

shyamjvs on Aug 9, 2017