Detox: Detox v18/v19 crashes sometimes on iOS in DetoxSync

Description

When updating from v17.latest of Detox to v18.latest I started seeing flakiness in the react-native-firebase e2e tests.

This does not happen all the time, it only happens sometimes. It looks like a bit of a race that some object destruct code is losing sometimes?

  • I have tested this issue on the latest Detox release and it still reproduces

Reproduction

Provide the steps necessary to reproduce the issue. If you are seeing a regression, try to provide the last known version where the issue did not reproduce.

  1. The react-native-firebase test suite is fully open source - instructions here https://github.com/invertase/react-native-firebase/blob/master/tests/README.md
  2. git clone git@github.com:invertase/react-native-firebase.git
  3. yarn && yarn tests:ios:pod:install && yarn tests:ios:build
  4. In a few terminals: yarn tests:packager:jet and yarn tests:emulator:start and yarn tests:ios:test

Sometimes (not every time, maybe 10% of the time? 20% fo the time?) it will crash with the stack below.

Expected behavior

It rolls through all our tests. For the purposes of this issue, it runs at all 😃 - if it is going to crash, it will do so with no tests passing, if it starts running and a single test passes, they will all roll through It usually crashes at startup but sometimes will make it through a handful of tests first.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • Detox: v18.22.0
  • React Native: 0.66.0-rc.4 (about to be 0.66.0 with no changes)
  • Node: v14.17.6
  • Device: apple simulator
  • Xcode: 12.5.1 (so I can avoid ios15 issues #2895 which are in progress separately)
  • iOS: 14.5 (to avoid issues with ios15 #2895)
  • macOS: macOS 11.6 (Apple Silicon M1 or Intel Mac - I’ve seen it on both)

Logs

It either works through all tests, or sometime shortly after app launch it crashes like this:


detox[98942] ERROR: [WS_ERROR] The app has crashed, see the details below:

The app has crashed, see the details below:

Signal 11 was raised
(
        0   Detox                               0x00000001104c5a15 +[NSThread(DetoxUtils) dtx_demangledCallStackSymbols] + 37
        1   Detox                               0x00000001104c8e10 __DTXHandleCrash + 464
        2   Detox                               0x00000001104c9555 __DTXHandleSignal + 59
        3   libsystem_platform.dylib            0x00007fff60335d7d _sigtramp + 29
        4   ???                                 0x00007f8fdf920050 0x0 + 140255907938384
        5   CoreFoundation                      0x0000000114170833 -[__NSDictionaryM dealloc] + 128
        6   libobjc.A.dylib                     0x0000000113f3b604 objc_object::sidetable_release(bool, bool) + 174
        7   libobjc.A.dylib                     0x0000000113f378ad _object_remove_assocations + 562
        8   libobjc.A.dylib                     0x0000000113f34c02 objc_destructInstance + 84
        9   libobjc.A.dylib                     0x0000000113f3ad88 -[NSObject dealloc] + 21
        10  libobjc.A.dylib                     0x0000000113f3b604 objc_object::sidetable_release(bool, bool) + 174
        11  CFNetwork                           0x0000000111038610 _CFNetworkHTTPConnectionCacheSetLimit + 162696
        12  CFNetwork                           0x0000000110e81183 CFNetwork + 24963
        13  DetoxSync                           0x0000000146cf0220 ____detox_sync_dispatch_wrapper_block_invoke + 23
        14  libdispatch.dylib                   0x00000001148c370d _dispatch_call_block_and_release + 12
        15  libdispatch.dylib                   0x00000001148c48df _dispatch_client_callout + 8
        16  libdispatch.dylib                   0x00000001148cae15 _dispatch_lane_serial_drain + 715
        17  libdispatch.dylib                   0x00000001148cb9c3 _dispatch_lane_invoke + 455
        18  libdispatch.dylib                   0x00000001148d5f81 _dispatch_workloop_worker_thread + 772
        19  libsystem_pthread.dylib             0x00007fff6034045d _pthread_wqthread + 314
        20  libsystem_pthread.dylib             0x00007fff6033f42f start_wqthread + 15
)

If you are experiencing a timeout in your test

If you are seeing a Detox build problem (e.g. during npm install, not detox build)

  • I am providing the npm install log below:

Device and verbose Detox logs

  • I have run my tests using the --loglevel trace argument and am providing the verbose log below:

Reproducible Demo

In case of a bug or a crash please add an example forking from the DetoxTemplate (follow the guidelines there) which reproduce the issue and ready to clone. Add to the DetoxTemplate (After you fork it), the minimal things which required to reproduce the issue (3rd party libraries / e2e tests).

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 4
  • Comments: 27 (14 by maintainers)

Commits related to this issue

Most upvoted comments

I just opened a related issue on firebase-ios-sdk repo: https://github.com/firebase/firebase-ios-sdk/issues/9083.

Also, this seems to be an issue we may be able to solve from DetoxSync’s end. I’ll investigate this a bit more but the general direction is to dispose the runtime-generated class that DetoxSync created before Firebase tries to dispose their generated class (the superclass of it), which will prevent the crash of calling objc_disposeClassPair() on a class before disposing its subclasses. See Apple docs:

Do not call this function if instances of the cls class or any subclass exist.

At the moment, the quickest solution I can offer is to disable Firebase Performance when Detox Tests are running.

Was previous having this same stacktrace as https://github.com/wix/Detox/issues/3000#issuecomment-992731382

Tried @mikehardy’s solution (but beware of the typo in the env variable!) Adding SIMCTL_CHILD_NSZombieEnabled=YES to my test script resolved the crash for me (to be determined if it’s consistent or just reduces crash rate as indicated in another thread I’m having trouble finding right now)

Thanks @mikehardy for your investigation & work to uncover this fix!

Interesting - well, I need it to work with firebase performance integrated of course 😃, even if a hypervisor is only a very strong “want” vs a need. It appears that I need to run a test on bare metal without firebase performance to see if I can at least reproduce clean runs. Then I suppose to actually make this more easily debuggable for someone with appropriate skills (that being, solid Objective-C skills which I lack unfortunately) I’ll need to fork here and add a test with firebase performance in it. Thanks for the info @shamilovtim

#3135 was merged, it solve (with a workaround) the issue of DetoxSync crashes when Firebase/Performance is integrated, it was tested on the example apps that reproduced this issue.

We will release a version with this change soon.

Once you have updated a version (to the latest/next version), if any of you are still having Signal 11 crashes on DetoxSync, please open a new bug report with the required details.

Huge thanks @mikehardy for your support and the assistance in solving this problem.

Probably not? Per my bio I attempt to reverse entropy as a full-time thing, but there always seems to be more of it 😉. Most likely that is unnecessary, at least I’m not aware of any reason why we reach into .bin, and I’m aware than in a yarn v2+ future that’s non-functional

Found this disclaimer in the Firebase performance SDK docs: Screen Shot 2021-12-12 at 11 37 03 AM

Validates theories that swizzling is conflicting

Detox: v18.22.1 React Native: 0.64.33 Node: v14.15.4 Device: apple simulator Xcode: 12.5.1 iOS: 14.5

This occurs for our team in CircleCI only:

DetoxRuntimeError: The pending request #-1000 ("isReady") has been rejected due to the following error:

    The app has crashed, see the details below:

    Signal 11 was raised
    (
    	0   Detox                               0x0000000110dc66c5 +[NSThread(DetoxUtils) dtx_demangledCallStackSymbols] + 37
    	1   Detox                               0x0000000110dc9250 __DTXHandleCrash + 464
    	2   Detox                               0x0000000110dc9991 __DTXHandleSignal + 59
    	3   libsystem_platform.dylib            0x000000011cc55d7d _sigtramp + 29
    	4   ???                                 0x0000000000000000 0x0 + 0
    	5   DetoxSync                           0x0000000152985ca8 +[DTXRunLoopSyncResource _existingSyncResourceWithRunLoop:clear:] + 120
    	6   DetoxSync                           0x0000000152980734 +[DTXSyncManager _untrackCFRunLoop:] + 45
    	7   DetoxSync                           0x00000001529806d5 +[DTXSyncManager untrackCFRunLoop:] + 75
    	8   DetoxSync                           0x000000015297ca0b swz_runRunLoopThread + 116
    	9   Foundation                          0x00000001117458a9 __NSThread__start__ + 1068
    	10  libsystem_pthread.dylib             0x000000011cc7c8fc _pthread_start + 224
    	11  libsystem_pthread.dylib             0x000000011cc78443 thread_start + 15
    )

@amq I see no instance of the string DetoxSync in your stack traces in any location. I believe your crash is unrelated, and is likely only showing up with Detox at all because Detox registers itself as a general crash handler in the system so all crashes go through it. Your crash does not actually look Detox related at all.

Here is my comment from the other thread: https://github.com/wix/Detox/issues/2802#issuecomment-953959588

I did extensive load tests on my own machine with 100% CPU usage on bare metal and could not reproduce the issue. However, the moment Detox runs under a VM (which is 16GB mem, 8*vCPU and running at 5% load), the issue appears. What I can conclude is that this is some sort of incompatibility between Detox and a hypervisor. Detox + virtualization, not performance, seems to cause the error.