webdriverio: Why does not Webdriverio report an error in beforeAll with Jasmine?

I use with Jasmine. When an error occur in beforeAll, WebdriverIO does not report the error, but exit code is 1. I expect it to report error and test failed. When an error occur in ‘beforeEach’, WebdriverIO report it and the test failed.

example code

describe('sample', () => {
  beforeAll(() => {
    throw new Error("ERROR")
  })

  it('sample', () => {
    browser.url('http://www.google.com')
  })
})

Environment (please complete the following information):

**WebdriverIO version: 4.14.0
Mode: WDIO Testrunner
**If WDIO Testrunner, running sync/async:“” sync
**Node.js version: 8.6.0
NPM version: 6.4.1
Browser name and version: Chrome 70
**Platform name and version: mac OS 10.13.6
Additional wdio packages used (if applicable): [e.g. @wdio/spec reporter, @wdio/selenium-standalone service]
- “chromedriver”: “2.44.0”,
- “wdio-chromedriver-service”: “0.1.3”,
- “wdio-jasmine-framework”: “0.3.7”,

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 21 (13 by maintainers)

Most upvoted comments

I have a possible patch for this, which I can PR for initial eyes after a little cleanup. It hooks into Jasmine.Suite’s beforeAll, afterAll and onException methods and emits hook:start and hook:end events at appropriate times. But it still leads to some reporting that’s pretty counter-intuitive if you are used to Mocha.

@nicholasbailey Please advise if you can share that patch/MR. I would be happy to take it from there, wrap it up and contribute an MR based on your work if you have no objections. This is quite nasty bug that makes people make quite an ugly workarounds to overcome it. Thanks!

mike-d-davydov on May 26, 2019

The fact that beforeAll swallows all errors caused quite a bit of confusion / frustration when I upgraded from v4 to v5. I think it would be really useful to fix this issue.

mo on Feb 14, 2019

Has there been any update on this issue?

VKnightUT on Apr 22, 2019

I think I’m ok with them continuing execution. What I think is the current (my experience is mostly in v4, but I think it also exists in v5) is that errors in beforeAll, afterAll, etc are completely swallowed and don’t appear in logs at all.

jrobinson01 on Feb 14, 2019

Digging into this over the last few days, it looks like part of the problem is that at least spec-reporter is only looking for failures on hook:end and test:failed. An exception beforeEach will trigger a test:failed event in wdio-jasmine-framework, but beforeAll will not. wdio-jasmine-framework (unlike wdio-mocha-framework) never emits a hook:end event, because Jasmine’s reporting interface doesn’t expose the start and end of hooks.

A further complication - Jasmine catches the exception in beforeAll and doesn’t propagate it immediately. So we can’t even wrap beforeAll and emit a hook:end event after it is called, because the exception won’t be available

I can imagine a couple of solutions here:

We could add a top-level notion of a suite having errors of its own, above and beyond hook and test errors. This would be a lot of work.
We could hook into Jasmine’s Suite.onException, and use that to trigger appropriate hook:end events. The downside here is that hook:end wouldn’t be emitted correctly if there were no exception.
We could queue all events for a suite and emit them after the suite actually runs. We can then use the data passed in suiteDone to figure out what errors occurred in any beforeAll and afterAll hooks.

In a vacuum, I’d do the third, but I don’t know the codebase/design choices as well as some other folks here.

nicholasbailey on Feb 13, 2019

Can you do the same fix in #2861 in wdio-mocha-framework to wdio-jasmine-framework to see if it can fix this as well?

No, that fix was related to Mocha

christian-bromann on Dec 5, 2018

@issei126 I tried replacing jasmine with mocha and it didn’t have this issue. That tells me that it’s a wdio-jasmine-framework issue rather than wdio. Read through the code of wdio-jasmine-framework and compare it with wdio-mocha-framework now to see if I can find anything.

xcao-riffyn on Dec 4, 2018