cypress-cucumber-preprocessor: After method is not being called if one of my steps failed

Hello Łukasz!

The problem with using Before to clear the state is that our application has a lot of complexity, relations between entities etc. Our environmets are even deployed pre-poluated in order to execute tests as our applications is on top of three other really complex ones.

“The only times you ever need to clean up state, is if the operations that one test runs affects another test downstream. In only those cases do you need state cleanup.” this is basically the case for all our tests and we can’t rely on order of execution since we would like to run tests in parallel. If we have to create one clean function it will have to take care of every possible failure during any of the steps, because creation and management of a given object often relies on at least a couple of more created in advance during the before phase using API calls.

I see the benefits of running the cleanup during the Before phase, but unfortunately I don’t think it is sustainable for large scale applications. If someone ran into the same problem, please share a possible solution 😃

I haven’t looked at that one at all to be honest but there are questions from Badeball there that seem unanswered, so I thought about directing my attention to other areas for now.

@lgandecki Hey Łukasz,

What are the unanswered questions? I don’t see comments from Badeball on this thread. This is a show stopper for us, I know you suggest that we call something that will restore our system to its initial state but that is not feasible in our case. Our application is run on top of 8-16 servers so we don’t have a reliable/timely way to restore all the applications that we’re built on top of. Before our suites are run, each suite has to create 8-16 VMs that our app runs on and then make a number of API calls (different per suite) to configure them for different scenarios.

We are not a single server app that can easily have its db re-seeded before each test, but we do know what each test modifies, so undoing what a test has done is the only way for us to be able to reliably run tests over and over after they fail. Right now we have to start another job to get a fresh testbed, which can take up to two hours making it really hard to work on tests that are currently failing.

I don’t understand why you’d offer an @After() and then not be willing to take a PR that makes it work. Are you saying that if an app cannot restore its state on a @Before() that Cypress may not the right tool for testing it?

It’s frustrating to have you say

Most importantly - Yes we would definitely take a PR with a fix. You can look at resolveStepDefinition.js hookRegistry.js.

And then have the PR rejected without any real explanation.

Can you do the cleanup on Before? https://docs.cypress.io/guides/references/best-practices.html#Using-after-or-afterEach-hooks

this is probably fine as a workaround. but it’s not very reasonable to have a feature file perform the teardown of say persisted data that have nothing to do with the feature being tested. ideally teardown of these objects would occur in context (ie in the feature file that created them) – preferably in an After that always runs.

Most importantly - Yes we would definitely take a PR with a fix.
You can look at resolveStepDefinition.js hookRegistry.js.

as for a philosophical discussion… 😃

I’m still not sure why can’t you reset the state before the tests though. I imagine a before step that brings the state of the app to the same “fresh” state. You don’t need to worry about removing users you added. The problem with arbitrary after steps that clean things that you’ve done over the test might be problematic - your test might actually change state in ways you didn’t plan for and subsequently breaking other tests. Worst case scenario if your tests run in random order (for example because of parallelization), those errors will be random and extremely hard to track.

In all honesty we do have a large legacy project where we do use after hook to clean the DB changes (not with cypress, that was years before cypress was made public), but we had to program in a “tracker” that hooked into our database, recorded all changes, and in the end reverted them. Without that the tests were flaky. Now that I think about it - we could as well just run the same logic in before step 😃