ember-test-helpers: setupOnError not working

I got fed up with this long-standing issue, so I decided to invest a bit of time and do some detective work. 🕵️😄

This occurs, when an error is thrown outside of the Ember run loop. setupOnerror works by hooking into Ember.onerror, which in turn is called by Backburner (Ember’s backing run loop implementation) when a task in the run loop throws.

The error then propagates to the global window.onerror handler or window.onunhandledrejection, if it’s async.

It still shows up as a global failure in your QUnit test, because QUnit catches these via QUnit.onUncaughtException, like @snewcomer correctly pointed out in the issue that you linked: https://github.com/emberjs/ember-qunit/issues/592#issuecomment-971106445

Running outside of a run loop can easily happen by accident, for instance when you manually add an event listener to an element and don’t wrap the listener in run().

import { run } from '@ember/runloop';

element.addEventListener('click', () => {
  run(() => {
    throw Error('caught by `setupOnerror`');
  });
});

element.addEventListener('click', () => {
  throw Error('NOT caught by `setupOnerror`');
})

And in fact, if you change your check action to throw the error inside a run loop, the test passes!

import Component from '@glimmer/component';
import { action } from '@ember/object';
import { run } from '@ember/runloop';

export default class MyComponent extends Component {
  @action
  check() {
    run(() => {
      throw new Error('check failed');
    });
  }
}

Now the Ember Guides claim that

For basic Ember app development scenarios, you don’t need to understand the run loop or use it directly. All common paths are paved nicely for you and don’t require working with the run loop.

This would lead me to believe that using {{on "click" this.check}} and the click test helper should just work™ out of the box. So why doesn’t it?

Ember transparently uses Glimmer’s {{on}} modifier. Glimmer itself is not integrated with Backburner nor the Ember runloop. It registers the passed in listener as-is without wrapping it in run() first.

I would consider this is a bug, because it violates the promise made in the Ember Guides that you don’t need to concern yourself with run when using Ember’s standard tools. It doesn’t really matter for real-world apps, because Ember will helpfully wrap calls that schedule tasks into a run loop queue into an implicit autorun loop, if they are not called from inside a run loop. The only negative effect of this is slightly worse performance.

This only really becomes an issue in tests, because Ember.onerror is not set up to handle errors thrown outside of a run loop, like explained above.

But the trouble doesn’t stop there. Regular “native” DOM events initiated by a user action / the browser are handled asynchronously as opposed to synthetic events, i.e. events triggered manually via dispatchEvent:

Unlike “native” events, which are fired by the browser and invoke event handlers asynchronously via the event loop, dispatchEvent() invokes event handlers synchronously. All applicable event handlers are called and return before dispatchEvent() returns.

click and any other DOM test helpers firing events use dispatchEvent via fireEvent. And since dispatchEvent triggers the event listeners synchronously, one would hope that, even if setupOnerror doesn’t work, the error would at least propagate through to click:

try {
  await click('.some-element-that-throws-on-click');
} catch (error) {
  console.log(error); // This should log the error from the click handler.
}

Alas, that also doesn’t work: #310, #453, #440

But why? Even though dispatchEvent calls all listeners synchronously, it doesn’t propagate any of the errors they might throw. So there again, the only option is to handle them in a global error handler. As of now, that’s window.onerror, which QUnit hooks into and then fails the test with the out of band global failure, like explained above.

Ideally though, the handler would be wrapped in a run loop, so that Ember.onerror can catch it instead. That would allow for handling it via setupOnerror, but the click helper still wouldn’t throw, because it doesn’t hook into Ember.onerror to watch for errors thrown during the execution of the event listeners.

Made an example: https://github.com/amk221/-ember-unhappy-path-test/pull/1

There is another example of how to do this here: https://github.com/qunitjs/qunit/issues/1419#issuecomment-561739486 – albeit, not contextually ergonomic (nor does it actually work – as QUnit’s properties are readonly (only QUnit.config is mutable)).

So, I’ve opened this issue on the QUnit repo, where I think we should figure out some solution for QUnit v3. https://github.com/qunitjs/qunit/issues/1736

Maybe it’s qunit that needs to provide setupOnError / resetOnError (or similar) for handling this situation.

@amk221 Looks like this is a known issue, there is some discussion in the latter part of this thread in the ember-qunit repo, which it looks like you were part of but there has been more discussion since your last comment there. Seems to be currently unresolved.