realm-java: Samsung Tab 3 Lite 7 crashes on Realm 2.0.1 and 2.0.2

By aligning compiler options (https://github.com/realm/realm-core/pull/2428), I have not been able to reproduce the crash.

Realm Java version 3.1.1 is online and contains the fix.

Since @diegomontoya had done the hard work of compiling, I decided to search for bugs related to optimization flags. I found https://code.google.com/p/android/issues/detail?id=81692. The bug reported in for a similar device, and the bug is that memmove is broken when optimization is enabled. The bug is consistent with work of @diegomontoya and my hypothesis that memcpy and/or memmove are the guilty piece of code.

I need to investigate a bit further before I can conclude if we have found the cause.

To anyone who can reproduce this issue:

We made a fix #4402 which works for the original test on our SM-T111. But it crashes when it comes with a lot memory pressure. Since the device itself is quite unstable, it crashes even in other apps/system apps, I’d like to believe there are some other system bugs on the device.

So, we want to verify the fix on some other devices. If you have a device which can reproduce this issue before, please try the fix. There are two ways to try it:

1. Compile realm-java from #4402 branch, and use it through mavenLocal. or
2. I made a desk build which can be found in the attached zip file.

• unzip it to $HOME/.m2/repository
• change the realm version in gradle classpath "io.realm:realm-gradle-plugin:3.0.1-SNAPSHOT"
• add mavenLocal in top level build.gradle like:

buildscript {
//...
    repositories {
        //...
        mavenLocal()
    }
//...
}

allprojects {
    repositories {
        //...
        mavenLocal()
    }
}

If your device has this problem, with the snapshot, you should be able to see: memmove is broken on this device. switch to the builtin implementation. in the logcat.

Then build the apk to see if it works with the fix. Thanks.

realm-java-maven-local-3.0.1-0.zip

We have merged #4402 and @beeender is currently preparing version 3.1.1. I thank everybody for helping us with bug reports, testing custom builds/snapshots and suggestions. The many builds by @diegomontoya helped us in the understanding, and @jonasbark has tested builds in his test lab.

I have been debugging most days this week without coming to a conclusion. The crash happens in the following instruction: STRB r4, [r1], #1. The register r1 contains an invalid address. My experiments this week indicate that we are talking about a race condition.

The crash happens in a strange place:

    frame #0: 0x5e61ea68 librealm-jni.so`realm::ArrayString::set(unsigned int, realm::StringData) + 162
    frame #1: 0x5e640c1e librealm-jni.so`realm::Group::do_get_or_add_table(realm::StringData, bool (*)(realm::Spec const&), void (*)(realm::Table&), bool*) + 154
    frame #2: 0x5e5f8570 librealm-jni.so`(anonymous namespace)::create_metadata_tables(realm::Group&) + 152
    frame #3: 0x5e5f86a4 librealm-jni.so`realm::ObjectStore::set_schema_version(realm::Group&, unsigned long long) + 12
    frame #4: 0x5e5bb50c librealm-jni.so`Java_io_realm_internal_SharedRealm_nativeSetVersion + 292

Using an in-memory Realm still gives the crash.

I have written a blog post about this bug: https://news.realm.io/news/when-memmove-fails

@kneth I suspect the bug is related to the -Os flag which implies -O2 but some options disabled. Based on our tests with the 1.5.1 core on Realm v1 branch, change the compile flags, removing flto option for example, has serious stability issues with samsung 4.2.2 devices. However changing to -O1 is stable.

The following results only applies to samsung 4.2.2 device we have. Other devices are fine with any compile flag combo.

original:
CFLAGS_OPTIM = -Os -flto -ffunction-sections -fdata-sections -DNDEBUG -fsigned-char -fvisibility=hidden

unstable:
CFLAGS_OPTIM = -Os -ffunction-sections -fdata-sections -DNDEBUG -fsigned-char -fvisibility=hidden

unstable:
CFLAGS_OPTIM = -O2 -ffunction-sections -fdata-sections -DNDEBUG -fsigned-char -fvisibility=hidden

unstable:
CFLAGS_OPTIM = -O2  -DNDEBUG -fsigned-char -fvisibility=hidden

stable:
CFLAGS_OPTIM = -O1 -DNDEBUG -fsigned-char -fvisibility=hidden

Integrated the snapshot as classpath "io.realm:realm-gradle-plugin:2.4.0-SNAPSHOT" Unfortunately the app still crashes on a Galaxy S3 mini:

02-10 16:15:02.593 10641-10641/link.thismo.app A/libc: Fatal signal 11 (SIGSEGV) at 0x5ab96ff6 (code=2), thread 10641 (link.thismo.app) 02-10 16:15:02.601 10641-10713/link.thismo.app I/dalvikvm: JNI ERROR (app bug): accessed deleted local reference 0x23400005 02-10 16:15:02.601 10641-10713/link.thismo.app E/dalvikvm: VM aborting 02-10 16:15:02.601 10641-10713/link.thismo.app A/libc: Fatal signal 11 (SIGSEGV) at 0xdeadd00d (code=1), thread 10713 (AnalyticsWorker) 02-10 16:15:10.054 10641-10647/link.thismo.app A/libc: @@@ ABORTING: LIBC: HEAP MEMORY CORRUPTION IN dlmalloc

Swapping the order of how the two metadata tables are created might solve the issue now. But I fear that it is masking the real bug - and I want to find it!

I appreciate wanting to find the source of the bug, but I’m with @Zhuinden on this one. Patching it for everyone in a release and continuing to investigate would be ideal from my perspective.

I have been debugging on this issue today. As already mention in my previous comment, I also experience crashes using a SM T111 device. Unfortunately the device is a bit unstable (random reboots even when I am not using it). The IntroExample crashes and sometimes the GC test suite.

Looking at core’s unit tests, I haven’t seen any crashes. Going back to previous core versions did give me anything.

The reported crashes (and mine in the IntroExample) indicate an issue with setVersion(). But simpler unit tests e.g., schemaIndexCacheIsUpdatedAfterSchemaChange do not crash.

I’ll keep debugging.

I am thinking about maybe there is another approach to solve this:

1. When the app starts, to testing of memmove and set function ptr workaround_memmove = workaround_needed ? &fixed_memmove : &org_memmovetrue;.
2. use gcc’s -wrap feature to wrap the memmove:

extern "C"
{
void* org_memmove(void *dest, const void *src, size_t n)
{
    return __real_memmove(dest, src, n);
}

void* fixed_memmove()
{
    return __real_memmove(dest, src, n) - n;
}

void *__wrap_memmove(void *dest, const void *src, size_t n)
{
    return (*workaround_memmove)(dest, src, n);
}
}

since it is using function pointer here, there should be no performance penalty at all for the devices which don’t have this problem.

I still get crashes when building everything with -fno-builtin-memove and -fno-builtin-memcpy. Next step is to try my own copy_backward.

@kneth @Konsumierer created the issue: https://github.com/realm/realm-java/issues/4177

@PiotrWpl Unfortunately Realm Core + Realm Sync were not released prior to Realm Java 2.3.1. It’s likely to hit Realm Java soon (at least as a snapshot release).

I have created https://github.com/realm/realm-java/pull/4067 which is a temporarily solution. Once merged, it should be available as a snapshot release. Please give it a try and report back. I am sure that this is not the proper fix but if other crashes surface from this solution, we might use that information to build a better hypothesis of that the root cause is.

I update backtrace in my FC case: Samsung device T111, Android 4.2.2, Realm version 2.2.0, Chipset: Marvell PXA986, ISA Supported: ARMv7, CPU Structure: RISC

01-10 13:53:34.550   103   103 I DEBUG   : backtrace:
01-10 13:53:34.550   103   103 I DEBUG   :     #00  pc 00086598  /data/app-lib/com.project-1/librealm-jni.so
01-10 13:53:34.550   103   103 I DEBUG   :     #01  pc 000a9d37  /data/app-lib/com.project-1/librealm-jni.so
01-10 13:53:34.550   103   103 I DEBUG   :     #02  pc 0005ec8d  /data/app-lib/com.project-1/librealm-jni.so

After using addr2line tool to revert backtrace:

std::system_error::system_error(int, std::error_category const&)
??:?
std::system_error::system_error(int, std::error_category const&)
??:?
realm::ObjectSchemaValidationException* std::__uninitialized_copy<false>::__uninit_copy<realm::ObjectSchemaValidationException*, realm::ObjectSchemaValidationException*>(realm::ObjectSchemaValidationException*, realm::ObjectSchemaValidationException*, realm::ObjectSchemaValidationException*)
??:?

I will post later if have any update.

@vadimbryl Please read my previous comment. We haven’t really had any progress.

The 2.4.0-SNAPSHOT is ready. Please try it out, and provide feedback. You can see in README.md how to use a snapshot release.

@kneth is this “fix” a part of today’s release Realm 2.3.1 for Android?

I am currently testing various combination of optimization flags, operating systems and compiler (only C++, not Android or Java).

EDIT:

I have been trying with the following test in Realm Core:

TEST(LangBindHelper_Memmove)
{
    char *array = strdup("Foobar");
    void *ptr = memmove(array + 1, array, sizeof("Foobar") - 2);
    CHECK(ptr == array + 1);
}

It is basically the test found in https://code.google.com/p/android/issues/detail?id=81692, and it testing the basic behaviour of memmove. The reason for testing memmove is that we have seen the crash in std::copy_backward which typically calls memmove (see for example line 339 in http://www.fifi.org/doc/gij-3.0/libstdc++/html_user/stl__algobase_8h-source.html).

The test passes on MacOS (clang with -O3) and OnePlus One (Android 6.0.1, gcc 4.9 with -Os). But the test fails on Samsung Galaxy Tab 3 (SM-T111, Android 4.2.2, gcc 4.9 with -Os).

The fix in https://bugreports.qt.io/browse/QTBUG-34984 (adding -fno-builtin-memmove) did not work for me, and I will continue to explore other possibilities (including implementing our own version of copy_backward).

Just now, I have tried to build from master (almost identical to 2.4.0-SNAPSHOT) and run our introExample on Samsung Galaxy Tab 3 (SM-T111) and Samsung Galaxy S 3 mini (I8200). I have no crashes.

@jonasbark @Konsumierer Can you share either an apk or source code so I can try myself? If you don’t wish to share it in public, please send it to help@realm.io.

Hey guys, could you please give me a short status update? Is there a fix in sight? I need to decide whether to go back to 1.x, replace Realm completely or wait for the fix in order to being able to rollout again. Thanks!

@cmelchior guys, it’s been 3 months and this bug is rather extreme from an enterprise standpoint, I think inverting the two lines of code isn’t such a bad idea after all

Updated: 10/01/2017 I tested on another Samsung device with PXA986 chipset: T210 - Galaxy Tab 3 (7.0), Android OS version 4.4.2 and 4.1.2 and it work fine.

Unfortunately on Samsung device farm don’t have T111 and I8200N: http://developer.samsung.com/remotetestlab/rtlDeviceList.action ++++++++++++++++++ Hi guys,

It seem I also got this issue on Samsung S3 mini LE and Samsung Tab 3 7 with realm version 2.2.0.

Base on report of user (Unfortunate i don’t have log) one common thing i can see on both device is they using Marvell chipset: PXA986, does it related with realm native library.

@kneth You mentioned a proper fix but is there any chance on a temporary fix or work-around we can use in the meanwhile? We are also having customers with Samsung S3 Mini devices with this issue and going back to an older version of realm would be a real pain since migration is not backward compatible.

@kneth We have users with Samsung Galaxy Tab 3 Lite and Samsung S3 mini VE. We published beta builds of our app with realm 3.0.0 and 2.4.0-snapshot, but it crashes. There is no any logs. App just closes without any messages.

Any news about this issue?

@finnschiermer has instrumented Realm Core to see if we could verify an issue with memcpy. His instrumentation is printf like debugging (writing to the Android log). After a number of experiments, we don’t see a crash when the instrumentation is enabled. But disabling the instrumentation, the tests crash. Apparently, our logging changes the memory layout in such a way, that the bug is not exposed.

I am almost 💯 % certain that @beeender was building it from that source - maybe with a unreleased core at the time.

We’ll soon push a new beta build of our app with realm 3.0.0. We have a user willing to try it with a Galaxy S3 Mini. I’ll update when I have the results.

Previous crashing device Samsung SM-G3818 (4.2.2) ran latest introExample test apk without issue.

Nyeh I think he just forgot to add the snapshot repository

buildscript {
    repositories {
        maven {
            url 'http://oss.jfrog.org/artifactory/oss-snapshot-local'
        }
    }
    dependencies {
        classpath "io.realm:realm-gradle-plugin:<version>-SNAPSHOT"
    }
}

allprojects {
    repositories {
        maven {
            url 'http://oss.jfrog.org/artifactory/oss-snapshot-local'
        }
    }
}

@brunozaranza Can you try with 2.4.0-SNAPSHOT?

@kneth, just ordered an I8200N on ebay. Will report back as soon as it gets here.

This has to be the strangest bug of the century.

Our latest release included an upgrade of Realm Core. As @diegomontoya notices, some changes in allocations is in that version. But that did not solve the issue (I know that @diegomontoya suggested to revert it but the crash happens with and without that particular commit).

Swapping the order of how the two metadata tables are created might solve the issue now. But I fear that it is masking the real bug - and I want to find it!

@kneth. I assume copy_on_write() is called on column expansion in core. Can you humor me and revert this commit and see if it helps? https://github.com/realm/realm-core/commit/13a4fe72c34c7f2dee828350ff7a1f18683a7f7d

I have had some progress today. When the Realm file is initially created, two tables are created (one for primary keys and one for metadata). At this point no other tables exist, and these are the first two tables added; the model classes are added later on. I can avoid the crash by swapping the order the two tables are created.

The crash occurs at realm::ArrayString::set(unsigned int, realm::StringData) i.e., deep in Realm Core when the table names are stored. Core will resize the width of the string column if required, so expanding from 2 to 8 bytes leads to the crash (adding the table with the longer table name first does not require an expansion of the column - and therefore no crash).

At this time, I can not sure what the proper fix is. Since it only happens at certain devices, I will have to get a deeper understanding on how these devices (and the Linux kernel and runtime environment) are different from “normal” devices.

I believe I have reproduced this one as well on my Samsung SIII mini armv7 GI-I8200 using realm version 2.2.1

It crashed on this particular function, not sure if the nativePtr is invalidated.

nativeSetVersion(nativePtr, schemaVersion);

I doubt that is the case here, since in that code path, the file has already been created and opened.