cluster-api: test: passed test suite contains failed test cases

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.]

example: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-capz-windows-dockershim/1417250279195152384

The job above has two failed test cases but it’s marked as passed:

Test started yesterday at 3:31 PM passed after 1h24m54s. (more info)

What did you expect to happen:

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

cluster-api-provider-azure version:
Kubernetes version: (use kubectl version):
OS (e.g. from /etc/os-release):

About this issue

Original URL
State: open
Created 3 years ago
Comments: 20 (18 by maintainers)

Most upvoted comments

Definitely still the case.

This can be reproduced ~ like this:

Run an upgrade test locally with a breakpoint before we run kubetest
Break the cluster (e.g. change the image in the CoreDNS Deployment to an invalid name)
Wait 10m (or add system-pods-startup-timeout: 1m to conformance.yaml
Step through the remaining code. I would expect us to copy the junit reports around but based on the code we don’t read and parse the results to fail the e2e test if there are any failures in the junit reports

cc @knabben (Just in case you’re looking for more work in a bit 😃, I can help you getting the local e2e test up & running)

sbueringer on Jan 3, 2023

If it does what we want, i don’t see why not.

randomvariable on Aug 20, 2021

Chat from Slack, looks like we’re going to have to parse the JUnit XML and figure out if something failed, which is pretty terrible.

Philosophically in the initial implementation, the current behaviour was intended given that to get any sort of result from conformance was a big success in itself. Taking the next step in making a failed conformance test fail the suite is reasonable.

randomvariable on Aug 19, 2021