realm-swift: mmap() failed: Cannot allocate memory size: 3221225472 offset: 0

Bug

I am currently doing as you suggested and I am still seeing this issue appearing in production for close to 10% of our users. This is also extremely hard to debug since I am unable to get a copy of the database from our users in the alpha.

I am running the shouldCompactOnLaunch as well as wrapping the writes in autoreleasepool so they aren’t kept in memory, as well as accessing the items one by one for processing before syncing to the server. I have tried all the steps outlined in several users issues that around the same problem.

Basically, I am using realm as an intermediate store for location data before sending up to the server, I went down the route of limiting the number of items choose to send up between sends but since they are lazily accessed this proves to be not the issue.

Goals

I would like to be able to simply read values that haven’t been pushed up since the last sync to our servers and store new values as they come in, to be synced in the future. I am not using realm server for this.

Expected Results

I expect to be able to read the data from the database with the predicate set to get the new location data that has been created since the last sync.

Actual Results

All I keep getting is this message that other have seen elsewhere.

mmap() failed: Cannot allocate memory size: 3221225472 offset: 0

Which then causes an entire application to crash. Which also seems quite high considering my shouldCompactOnLaunch code.

Steps to Reproduce

All I am doing is a simple read and the application crashes from what I gather it isn’t able to read from the database since it needs to move it into memory before the read and it cannot do so. I have tried the shouldCompactOnLaunch strategy as well as wrapping reads in autoreleasepool

Code Sample

Here is `shouldCompactOnLaunch` code

RLMRealmConfiguration *config = [RLMRealmConfiguration defaultConfiguration];
  config.shouldCompactOnLaunch = ^BOOL(NSUInteger totalBytes, NSUInteger usedBytes){
    // totalBytes refers to the size of the file on disk in bytes (data + free space)
    // usedBytes refers to the number of bytes used by data in the file
    
    // Compact if the file is over 10MB in size and less than 50% 'used'
    NSUInteger tenMB = 10 * 1024 * 1024;
    return (totalBytes > tenMB) && (usedBytes / totalBytes) < 0.5;
  };
  
  NSError *error = nil;
  // Realm is compacted on the first open if the configuration block conditions were met.
  RLMRealm *realm = [RLMRealm realmWithConfiguration:config error:&error];
  if (error) {
    // handle error compacting or opening Realm
    NSLog(@"%@", error);
  }

Here is read that is causing the crash

  NSDate *lastSync = [self getVisitAnchor];

  NSPredicate *_today = [NSPredicate predicateWithFormat:@"endTime >= %@", lastSync];
  RLMResults<Visit *> *visits = [Visit objectsWithPredicate:_today];
  
  NSDate *lastVisitTime = [visits lastObject].startTime;
  
  NSMutableArray *visitArray = [[NSMutableArray alloc] init];
  
  for (NSInteger i = 0; i < [visits count]; i++) {
    // create an array structure.
    NSDictionary *visitData = [self serializeRealmVisit:visits[i]];
    [visitArray addObject:visitData];
  }

Which is what triggers the crash, here is the associated stack trace

Fatal Exception: RLMException
0  CoreFoundation                 0x191706fe0 __exceptionPreprocess
1  libobjc.A.dylib                0x190168538 objc_exception_throw
2  Realm                          0x10076f908 (Missing)
3  Realm                          0x100750d34 (Missing)
4  Realm                          0x100751e3c (Missing)
5  Realm                          0x100751bcc (Missing)
6  Realm                          0x1007502e8 (Missing)
7  Realm                          0x1006e4a50 (Missing)
8  Application                      0x10010e308 -[LocationManager sendVisits] (LocationManager.m:225)
9  Application                      0x10010dd74 -[LocationManager sendData] (LocationManager.m:161)
10 Application                      0x10010ddb0 -[LocationManager startMonitoring] (LocationManager.m:166)
11 Application                      0x10010c9a4 -[AppDelegate startMonitoring] (AppDelegate.m:112)
12 Application                      0x10010c818 -[AppDelegate application:didFinishLaunchingWithOptions:] (AppDelegate.m:106)
13 UIKit                          0x1978a9e48 -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:]
14 UIKit                          0x197ab637c -[UIApplication _callInitializationDelegatesForMainScene:transitionContext:]
15 UIKit                          0x197abbe24 -[UIApplication _runWithMainScene:transitionContext:completion:]
16 UIKit                          0x197ad08b0 __84-[UIApplication _handleApplicationActivationWithScene:transitionContext:completion:]_block_invoke.3147
17 UIKit                          0x197ab90b8 -[UIApplication workspaceDidEndTransaction:]
18 FrontBoardServices             0x1932b0884 __FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__
19 FrontBoardServices             0x1932b06f0 -[FBSSerialQueue _performNext]
20 FrontBoardServices             0x1932b0aa0 -[FBSSerialQueue _performNextFromRunLoopSource]
21 CoreFoundation                 0x1916b542c __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
22 CoreFoundation                 0x1916b4d9c __CFRunLoopDoSources0
23 CoreFoundation                 0x1916b29a8 __CFRunLoopRun
24 CoreFoundation                 0x1915e2da4 CFRunLoopRunSpecific
25 UIKit                          0x1978a2fc8 -[UIApplication _run]
26 UIKit                          0x19789dc9c UIApplicationMain
27 Application                      0x1001199d4 main (main.m:16)
28 libdyld.dylib                  0x1905f159c start

Version of Realm and Tooling

Realm framework version: 3.0.2

Xcode version: 9.1

iOS/OSX version: 10, 11

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 11
Comments: 60 (18 by maintainers)

Most upvoted comments

I recently ran into this issue as well. Luckily I was able the reproduce the issue in the simulator. I followed the steps from #5931 but that didn’t show the issue. The memory debugger only showed one instance of a realm in memory (the one for the main thread) and I’d added autoreleasing to all background thread access so that was working fine.

I just started disabling subsystems in the app until the issue stopped happening. Once I narrowed it down to a part of the app, I was able to find a retain cycle that was holding onto an unused instance of ThreadSafeReference. In cases where the app was working correctly, the ThreadSafeReference would be used and there wouldn’t be an issue. But there was an error case where some objects weren’t cleaned up and that’s what caused it for me.

TL;DR Long lived and unused instances of ThreadSafeReference can cause large realm files.

CraigSiemens on Oct 22, 2018

Hi @bdash, so sorry for bringing this up, but I have general questions (that I think are relevant to this topic) regarding to your replies in this thread.

A little context: I’m also experiencing the same problem myself (cannot allocate memory size) and I’m currently reading through similar issues, hoping to better understand version pinning; its underlying root cause, and how to address this problem the right way. I am also aware of two solutions currently:

Compaction upon startup (or when there are zero Realm instances open),
Wrap Realm initialization in @autoreleasepool (or invalidate manually) when executed in background threads.

I’m trying to understand the conditions that cause version pinning in Realm, and how to properly debug them. For example, I’m very interested in one of your earlier replies where you stated in this comment:

The Realm file you shared shows signs of the version pinning problem I described. It looks to contain data from 14,687 versions, with the current version being 14,692. That indicates a version was pinned very early in the lifetime of the app. …

In the Realm browser, I’m not sure there’s an information that states how many version it holds… Would you mind sharing how to check how many versions a Realm file contains (to prove that it indicates version pinning)?

So far from what I’ve read, RLMRealm-s are instantiated once per thread, and stored (strongly?) in an NSDictionary (with the key specifying the thread identifier). I’m curious as to why, in background thread / dispatch_async block, wrapping RLMRealm initialization in @autoreleasepool will correctly invalidate (release?) the RLMRealm? Sorry if I got the concept wrong.

Lastly, I’m trying to reproduce the Realm file inflation by identifying background-thread-Realm-instantiations, but after I’ve identified the step and tried to reproduce it multiple times, the memory usage increased but the Realm file failed to inflate. I’ve read through https://realm.io/docs/objc/latest/#limitations-file-size, but I’m having trouble to find such case within my app (I did find the background Realm operations, but there are no “read some data from the Realm and then block the thread on a long-running operation”). Would you mind sharing a simple example that ensures a Realm file will significantly inflate?

Sorry if this is not the appropriate section to ask the questions. Please do let me know if I should open a new issue in Github (or maybe StackOverflow). Thanks very much for your help!

dvdchr on Feb 11, 2018

I just posted our “fix” to this issue here… https://stackoverflow.com/questions/42527836/realm-mmap-failed-cannot-allocate-memory-size/51695949#51695949

pangia on Aug 5, 2018

I recently got an inflated realm file with size 2.01GB from one of users. I opened my app with this inflated realm file on simulator. An inflated realm file was opened and compact process had been processed. Now compacted file size is 2.4MB.

Please add a function to compact an extra large realm file without crashing on realm device. Reproducing is very hard but it’s happening… so the way to fix and prevent is needed.

xiles on Mar 20, 2018

The fundamental problem that you need to solve is why the file is ending up so much larger than you expect. That’s what I’m attempting to help you with, and what the questions I’m asking you are about.

A Realm file contains multiple versions of data within it. Each RLMRealm instance is looking at a specific version of data in the file. When the RLMRealm refreshes, it advances to the latest version of the data within the file. On threads with runloops, this happens automatically on runloop iterations after changes were committed. On threads without runloops, this needs to be done manually unless the RLMRealm instance is deallocated after being used. When a write transaction is performed, Realm will attempt to reuse space used by older versions of the data within the Realm file. However, if a RLMRealm is referring to a given version of the data, that version is considered to be “pinned” and cannot be reused until that RLMRealm is refreshed otherwise it would cause the code using that RLMRealm to have an inconsistent view of the data.

If the Realm file is open on the main thread, but the runloop is not able to iterate normally for some reason (maybe there’s a long-running operation, maybe the runloop is operating in a non-default runloop mode, or maybe iOS doesn’t run the runloop when delivering location event updates to background apps? I dunno), then the RLMRealm instance on the main thread will not automatically refresh to the latest version of the data. This pins the old version of the data that it is looking at, forcing any write transactions that occur to grow the file until that old version becomes unpinned.

bdash on Nov 22, 2017

@bmunkholm @tgoyne - Is there any way to avoid this - we’re having very hard time because of this - Not able to answer our customers. Please provide any solutions for this.

punithbm on Aug 3, 2018

@myusuf3 please contact someone from Realm about this as I’m no longer involved with this project.

jpsim on Apr 9, 2018

So the current workaround I have found is putting in an expensive size check when the app launches and seeing if the RealmDB is above a certain size and truncating it. I have tried all the suggestions proposed by @bdash none of them solved the issue.

Things I have tried:

@autorelease around when I am writing to the database to decrease pinning.
Using the connection through a singleton so there is only one thread writing to the database.
Reworking my application structure likely not the issue here but that was what was suggested.

None of these worked so I have expensive truncation of the database to bring it down to size and let it fit in memory. There is no way I am generating 3GB of data in few hours of location data so something is pinning but I am not sure how pinning is happening above a size that is possible of happening. I linked a pretty much empty database that was sitting at over several GB.

I even managed to pull realm files from a few users and shared it here to no avail, hoping this sees more interest, I would love to remove the expensive removal call.

myusuf3 on Feb 21, 2018