ginkgo: continue after failure in ordered container

When running specs in an ordered container, ginkgo aborts after the first test failure:

  Spec skipped because an earlier spec in an ordered container failed

In my case, additional specs can run. I need them to run in a certain order not because there are dependencies between them (each spec is self-contained) but because it is more efficient (some setup code can get skipped) and some tests are more reliable when run first.

Is it possible to configure whether ordered containers fail fast?

Background

I am testing PMEM-CSI with ginkgo and started the migration to v2 there. I’m also helping with Kubernetes and csi-test, which need to be migrated first.

In PMEM-CSI, we have the problem that one of our dependencies, the Operator Lifecycle Manager (OLM) fails with random timeouts when the tests using it run after the test cluster has been used for a while. I don’t have a good explanation and clearly that shouldn’t happen. I suspect some performance degradation, but the only solution that I have found so far is to run the tests with OLM directly after bringing up a test cluster.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 17 (14 by maintainers)

Commits related to this issue

test: run OLM tests first, III Top-level nodes were still getting randomized (https://github.com/onsi/ginkgo/issues/966#issuecomment-1110046593). — committed to pohly/pmem-CSI by pohly 2 years ago

Most upvoted comments

However, the default behavior when running in series is to only shuffle the order of the outermost container nodes

This hint allowed me to solve my problem in PMEM-CSI: I moved the Describe calls from top-level into a Context and now they get executed in the order in which they are defined because --randomize-all is off.

As far as I am concerned, this issue can be closed.

pohly on Apr 27, 2022

hey @dmytro-zakharov Ginkgo lets you make the following trade-off:

Build independent specs that can be parallelized and randomized. This is the recommended approach and avoids using Ordered containers. You can amortize the cost of setup in a variety of ways (happy to dig into it further with you if interested) and by parallelizing your suite you can generally keep test performance reasonable.
Build ordered specs that allow you to express more complex flows as distinct Its. In effect, the specs in an Ordered container behave like one large It (in fact, people used to write these as one large It but asking for a BeforeAll became a common refrain in the community so I introduced Ordered containers to help people break up their specs into more bite-sized chunks).

Ginkgo’s opinion is that when a spec in an Ordered container fails the entire container is considered suspect and no subsequent specs will run. While I can imagine adding complexity to allow users to specify “but not this spec” I’m concerned that that will make the behavior of Ordered containers even harder to reason about. My preference is to guide users to prefer independent specs over correlated specs and to use Ordered only in the specific context it was intended for: breaking up large complex Its into smaller pieces to make the test a bit easier to read.

onsi on Apr 27, 2022