Detox: Detox v18/v19 crashes sometimes on iOS in DetoxSync
Description
When updating from v17.latest of Detox to v18.latest I started seeing flakiness in the react-native-firebase e2e tests.
This does not happen all the time, it only happens sometimes. It looks like a bit of a race that some object destruct code is losing sometimes?
- I have tested this issue on the latest Detox release and it still reproduces
Reproduction
Provide the steps necessary to reproduce the issue. If you are seeing a regression, try to provide the last known version where the issue did not reproduce.
- The react-native-firebase test suite is fully open source - instructions here https://github.com/invertase/react-native-firebase/blob/master/tests/README.md
git clone git@github.com:invertase/react-native-firebase.git
yarn && yarn tests:ios:pod:install && yarn tests:ios:build
- In a few terminals:
yarn tests:packager:jet
andyarn tests:emulator:start
andyarn tests:ios:test
Sometimes (not every time, maybe 10% of the time? 20% fo the time?) it will crash with the stack below.
Expected behavior
It rolls through all our tests. For the purposes of this issue, it runs at all 😃 - if it is going to crash, it will do so with no tests passing, if it starts running and a single test passes, they will all roll through It usually crashes at startup but sometimes will make it through a handful of tests first.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
- Detox: v18.22.0
- React Native: 0.66.0-rc.4 (about to be 0.66.0 with no changes)
- Node: v14.17.6
- Device: apple simulator
- Xcode: 12.5.1 (so I can avoid ios15 issues #2895 which are in progress separately)
- iOS: 14.5 (to avoid issues with ios15 #2895)
- macOS: macOS 11.6 (Apple Silicon M1 or Intel Mac - I’ve seen it on both)
Logs
It either works through all tests, or sometime shortly after app launch it crashes like this:
detox[98942] ERROR: [WS_ERROR] The app has crashed, see the details below:
The app has crashed, see the details below:
Signal 11 was raised
(
0 Detox 0x00000001104c5a15 +[NSThread(DetoxUtils) dtx_demangledCallStackSymbols] + 37
1 Detox 0x00000001104c8e10 __DTXHandleCrash + 464
2 Detox 0x00000001104c9555 __DTXHandleSignal + 59
3 libsystem_platform.dylib 0x00007fff60335d7d _sigtramp + 29
4 ??? 0x00007f8fdf920050 0x0 + 140255907938384
5 CoreFoundation 0x0000000114170833 -[__NSDictionaryM dealloc] + 128
6 libobjc.A.dylib 0x0000000113f3b604 objc_object::sidetable_release(bool, bool) + 174
7 libobjc.A.dylib 0x0000000113f378ad _object_remove_assocations + 562
8 libobjc.A.dylib 0x0000000113f34c02 objc_destructInstance + 84
9 libobjc.A.dylib 0x0000000113f3ad88 -[NSObject dealloc] + 21
10 libobjc.A.dylib 0x0000000113f3b604 objc_object::sidetable_release(bool, bool) + 174
11 CFNetwork 0x0000000111038610 _CFNetworkHTTPConnectionCacheSetLimit + 162696
12 CFNetwork 0x0000000110e81183 CFNetwork + 24963
13 DetoxSync 0x0000000146cf0220 ____detox_sync_dispatch_wrapper_block_invoke + 23
14 libdispatch.dylib 0x00000001148c370d _dispatch_call_block_and_release + 12
15 libdispatch.dylib 0x00000001148c48df _dispatch_client_callout + 8
16 libdispatch.dylib 0x00000001148cae15 _dispatch_lane_serial_drain + 715
17 libdispatch.dylib 0x00000001148cb9c3 _dispatch_lane_invoke + 455
18 libdispatch.dylib 0x00000001148d5f81 _dispatch_workloop_worker_thread + 772
19 libsystem_pthread.dylib 0x00007fff6034045d _pthread_wqthread + 314
20 libsystem_pthread.dylib 0x00007fff6033f42f start_wqthread + 15
)
If you are experiencing a timeout in your test
- I have followed the instructions under Identifying which synchronization mechanism causes us to wait too much, I have read synchronization debug documentation and am providing the relevant synchronization debug output below:
If you are seeing a Detox build problem (e.g. during npm install
, not detox build
)
- I am providing the
npm install
log below:
Device and verbose Detox logs
- I have run my tests using the
--loglevel trace
argument and am providing the verbose log below:
Reproducible Demo
In case of a bug or a crash please add an example forking from the DetoxTemplate (follow the guidelines there) which reproduce the issue and ready to clone. Add to the DetoxTemplate (After you fork it), the minimal things which required to reproduce the issue (3rd party libraries / e2e tests).
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 4
- Comments: 27 (14 by maintainers)
Commits related to this issue
- test(e2e): mitigate DetoxSync crashes root cause hypothesis is double-swizzling w/DetoxSync and performance mitigation hypothesis is to use new GUL swizzler feature to disable some object destruction... — committed to invertase/react-native-firebase by mikehardy 3 years ago
- test(e2e): mitigate DetoxSync crashes root cause hypothesis is double-swizzling w/DetoxSync and performance mitigation hypothesis is to use new GUL swizzler feature to disable some object destruction... — committed to invertase/react-native-firebase by mikehardy 3 years ago
- test(e2e): mitigate DetoxSync crashes root cause hypothesis is double-swizzling w/DetoxSync and performance mitigation hypothesis is to use new GUL swizzler feature to disable some object destruction... — committed to invertase/react-native-firebase by mikehardy 3 years ago
- iOS(AppleSimUtils): enable NSZombies on app launch. This workaround solves the issue described here: https://github.com/firebase/firebase-ios-sdk/issues/9083 And some of the crashes that was mention... — committed to asafkorem/Detox by asafkorem 3 years ago
- iOS(AppleSimUtils): enable NSZombies on app launch. This workaround solves the issue described here: https://github.com/firebase/firebase-ios-sdk/issues/9083 And some of the crashes that was mention... — committed to wix/Detox by asafkorem 3 years ago
I just opened a related issue on firebase-ios-sdk repo: https://github.com/firebase/firebase-ios-sdk/issues/9083.
Also, this seems to be an issue we may be able to solve from DetoxSync’s end. I’ll investigate this a bit more but the general direction is to dispose the runtime-generated class that DetoxSync created before Firebase tries to dispose their generated class (the superclass of it), which will prevent the crash of calling objc_disposeClassPair() on a class before disposing its subclasses. See Apple docs:
At the moment, the quickest solution I can offer is to disable Firebase Performance when Detox Tests are running.
Was previous having this same stacktrace as https://github.com/wix/Detox/issues/3000#issuecomment-992731382
Tried @mikehardy’s solution (but beware of the typo in the env variable!) Adding
SIMCTL_CHILD_NSZombieEnabled=YES
to my test script resolved the crash for me (to be determined if it’s consistent or just reduces crash rate as indicated in another thread I’m having trouble finding right now)Thanks @mikehardy for your investigation & work to uncover this fix!
Interesting - well, I need it to work with firebase performance integrated of course 😃, even if a hypervisor is only a very strong “want” vs a need. It appears that I need to run a test on bare metal without firebase performance to see if I can at least reproduce clean runs. Then I suppose to actually make this more easily debuggable for someone with appropriate skills (that being, solid Objective-C skills which I lack unfortunately) I’ll need to fork here and add a test with firebase performance in it. Thanks for the info @shamilovtim
#3135 was merged, it solve (with a workaround) the issue of DetoxSync crashes when Firebase/Performance is integrated, it was tested on the example apps that reproduced this issue.
We will release a version with this change soon.
Once you have updated a version (to the latest/next version), if any of you are still having Signal 11 crashes on DetoxSync, please open a new bug report with the required details.
Huge thanks @mikehardy for your support and the assistance in solving this problem.
Probably not? Per my bio I attempt to reverse entropy as a full-time thing, but there always seems to be more of it 😉. Most likely that is unnecessary, at least I’m not aware of any reason why we reach into .bin, and I’m aware than in a yarn v2+ future that’s non-functional
Found this disclaimer in the Firebase performance SDK docs:
Validates theories that swizzling is conflicting
Detox: v18.22.1 React Native: 0.64.33 Node: v14.15.4 Device: apple simulator Xcode: 12.5.1 iOS: 14.5
This occurs for our team in CircleCI only:
@amq I see no instance of the string DetoxSync in your stack traces in any location. I believe your crash is unrelated, and is likely only showing up with Detox at all because Detox registers itself as a general crash handler in the system so all crashes go through it. Your crash does not actually look Detox related at all.
Here is my comment from the other thread: https://github.com/wix/Detox/issues/2802#issuecomment-953959588
I did extensive load tests on my own machine with 100% CPU usage on bare metal and could not reproduce the issue. However, the moment Detox runs under a VM (which is 16GB mem, 8*vCPU and running at 5% load), the issue appears. What I can conclude is that this is some sort of incompatibility between Detox and a hypervisor. Detox + virtualization, not performance, seems to cause the error.