realm-swift: Uncontrolled increase of Realm database file size until app crash

Perhaps realm should provide it’s own method of performing background realm operations like CoreData e.g. realm.performBackgroundWrite { realm in realm.insert(object) }

This way you control over managing background realm instances (auto-releasing them), and getting the chance to commit background writes in batches.

Hi Mark – it is clear that a lot of people are having issues with uncontrolled expansion of the database file size. I hope the Realm team takes this seriously as we are going to have to migrate off Realm back to CoreData – and advise others to do the same – as there is zero excuse for a bug that turns an app into a complete lemon (due to crash on startup due to database file size) in production code.

We experience this (I guess it is the same, if not I will open a separate issue) also in the wild and I’m not sure what causes this. Our app stores data around ~20mb in the realm database at most, but we keep get reports of this:

fatal error: 'try!' expression unexpectedly raised an error: Error Domain=io.realm Code=9 "mmap() failed: Cannot allocate memory size: 3221225472 offset: 0" UserInfo={NSLocalizedDescription=mmap() failed: Cannot allocate memory size: 3221225472 offset: 0, Error Code=9}: file /Library/Caches/com.apple.xbs/Sources/swiftlang/swiftlang-800.0.63/src/swift/stdlib/public/core/ErrorType.swift, line 178

As this happens on app start and only for some users this is hard to track.

At the point of crash we have no open instances of realm and we try to do the following:

func compactRealm() {
        let defaultURL = realmConfig!.fileURL!
        let defaultParentURL = defaultURL.deletingLastPathComponent()
        let compactedURL = defaultParentURL.appendingPathComponent("default-compact.realm")

        if FileManager.default.fileExists(atPath: compactedURL.path) {
            try! FileManager.default.removeItem(at: compactedURL)
        }

        if FileManager.default.fileExists(atPath: defaultURL.path) {
            autoreleasepool {
                let realm = try! Realm(configuration: realmConfig) ## Here it crashes
                try! realm.writeCopy(toFile: compactedURL)
            }

            try! FileManager.default.removeItem(at: defaultURL)
            try! FileManager.default.moveItem(at: compactedURL, to: defaultURL)
        }
    }

The stack trace is also rather unintresting:

Crashed: com.apple.main-thread
0  libswiftCore.dylib             0x1028f4a18 _assertionFailed(StaticString, String, StaticString, UInt, flags : UInt32) -> Never + 164
1  libswiftCore.dylib             0x1029147c8 swift_unexpectedError_merged + 476
2  libswiftCore.dylib             0x1029145d0 swift_errorInMain + 26
3  OurApp                        0x100a61c8c static Database.(compactRealm() -> ()).(closure #1) (Database.swift:75)
4  libswiftObjectiveC.dylib       0x10306972c autoreleasepool<A> (invoking : () throws -> A) throws -> A + 68
5  OurApp                        0x100a6186c static Database.compactRealm() -> () (Database.swift:79)
6  OurApp                        0x100a613dc static Database.setup() -> () (Database.swift:60)
7  OurApp                        0x100ac4318 OurApp.init()
8  OurApp                        0x100ac4d80 static OurApp.setup() -> () (OurApp.swift)
9  OurApp                        0x1001b2d74 specialized AppDelegate.application(UIApplication, didFinishLaunchingWithOptions : [UIApplicationLaunchOptionsKey : Any]?) -> Bool (AppDelegate.swift)
10 OurApp                        0x1001ab970 @objc AppDelegate.application(UIApplication, didFinishLaunchingWithOptions : [UIApplicationLaunchOptionsKey : Any]?) -> Bool (AppDelegate.swift)
11 UIKit                          0x189f472dc -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:] + 380
12 UIKit                          0x18a153800 -[UIApplication _callInitializationDelegatesForMainScene:transitionContext:] + 3452
13 UIKit                          0x18a1592a8 -[UIApplication _runWithMainScene:transitionContext:completion:] + 1684
14 UIKit                          0x18a16dde0 __84-[UIApplication _handleApplicationActivationWithScene:transitionContext:completion:]_block_invoke.3151 + 48
15 UIKit                          0x18a15653c -[UIApplication workspaceDidEndTransaction:] + 168
16 FrontBoardServices             0x18594f884 __FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__ + 36
17 FrontBoardServices             0x18594f6f0 -[FBSSerialQueue _performNext] + 176
18 FrontBoardServices             0x18594faa0 -[FBSSerialQueue _performNextFromRunLoopSource] + 56
19 CoreFoundation                 0x183d55424 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24
20 CoreFoundation                 0x183d54d94 __CFRunLoopDoSources0 + 540
21 CoreFoundation                 0x183d529a0 __CFRunLoopRun + 744
22 CoreFoundation                 0x183c82d94 CFRunLoopRunSpecific + 424
23 UIKit                          0x189f4045c -[UIApplication _run] + 652
24 UIKit                          0x189f3b130 UIApplicationMain + 208
25 PACEApp                        0x1001112d0 main (AppDelegate.swift:32)
26 libdyld.dylib                  0x182c9159c start + 4

Fabric device stats tells us that there is around 2-5% of ram left when this happens but I think that is due to us trying to allocate GB’s of data…

Realm v2.4.4 Swift 3.0.2

I can’t reproduce this so sadly no sample project/… If you can give me a starting point I can try to pin this

What I also noticed is that we experience this on nearly all device types and the pretended size realm wants to mmap is only one of these four:

• Cannot allocate memory size: 3221225472
• Cannot allocate memory size: 2147483648
• Cannot allocate memory size: 1744830464
• Cannot allocate memory size: 2415919104

As the data we store is highly user dependent it would surprise me if these users have the same database sizes.

First, let me reiterate that large Realm files are likely due to holding old versions of data in a background thread for extended periods of time. Realms opened on threads without a runloop don’t auto-refresh, so you’ll need to take special care to wrap background Realm access in self-contained @autoreleasepool blocks that don’t leak any Realm-related types.

However, we understand that debugging these cases can be difficult, and even if you don’t leak Realms, file sizes can be larger than you’d expect.

So I’d like to encourage everyone who’s experienced large Realm file sizes to try out the new shouldCompactOnLaunch block on Realm.Configuration, available as of today’s 2.6 release: https://realm.io/docs/swift/latest/#compacting-realms

I wanted to provide a default implementation so this would be on-by-default, but struggled to find good one-size-fits-all heuristics. I think some form of “compact on launch” should be Realm’s default mode of operation. So I look forward to hearing what settings people end up using in practice and what ends up working best. Please share feedback from this so we can confidently enable something like this for everyone out of the box.

We do take it seriously, but we need more information in order for this to be actionable. Firstly, we need to understand the manner in which your app is using Realm, and we can’t usefully do that for multiple people in a single GitHub issue.

@numair, since you filed this issue we can happily gather information about your situation here. A few questions:

1. Can you reproduce this problem yourself at all, or has it only been reported by users?
2. Can you reproduce it on demand, or does it only appear to happen at random?
3. Can you describe how your application uses Realm with respect to multiple threads. For instance: a. Do you only use Realm on the main thread? b. Do you use Realm on dispatch queues? c. Do you use Realm on NSOperationQueues? d. Do you use Realm on background threads that you’ve created? e. On which threads / queues do you write to the Realm? f. On which threads / queues do you read from the Realm?
4. Is your Realm file only accessed by a single process at a time? Multiple process may be involved if you have multiple apps in an app group that share a Realm, or if you have application extensions that share a Realm with your main app.
5. How frequently do you write to the Realm?
6. Do you have multiple threads attempting to write to the Realm simultaneously?
7. Have you tested with the most recent version of Realm? v2.5.0 includes a fix for a problem that could lead to unexpected file size growth when you have write contention (i.e., multiple threads attempting to write to the Realm concurrently for an extended period of time).

For other people that are seeing a similar issue, please open a new GitHub issue and provide the information described in the issue template along with answers to these questions.

Explicitly calling invalidate() on Realm instances is one way to accomplish option 1.

(I was already compacting the Realm on every launch though).

Please be wary of doing this outside of Realm’s provided shouldCompactOnLaunch block, for the reasons outlined in the blog post announcing the feature: https://news.realm.io/news/realm-objc-swift-2.6#compact-on-launch

The instructions were followed but the issue only started to happen in realm 2.2 ~ 2.4 version.