vdexExtractor: Linux: compact_dex_converter fails with "Illegal instruction (core dumped)"

@superr Last try, built compact_dex_converter with amdfam10 optimization flags compact_dex_converter_amdfam10.zip

@superr Ok, i don’t know what cpu you use, but it seems that you can add -march and-mtune flags to linuxCflags in x86_linux_host.go. Set them to core2 and built new binary, maybe it will work. Have no more ideas how to fix this. compact_dex_converter_core2.zip

You can try to change compilation flags. But i’m not sure flags from what files used for host build. I recommend start from “/build/soog/cc/config” there you can find “x86_linux_host.go” open it and try to change

	// Extended cflags
	linuxX86Cflags = []string{
		"-msse3",
		"-mfpmath=sse",
		"-m32",
		"-march=prescott",
		"-D_FILE_OFFSET_BITS=64",
		"-D_LARGEFILE_SOURCE=1",
	}

you can try remove this

		"-march=prescott",
		"-msse3",
		"-mfpmath=sse",

if this won’t help you can open “global.go” and change commonGlobalCflags, for example “-O2” to “-O1”

@superr Open Android.bp and find:

art_cc_binary {
    name: "compact_dex_converter",
    defaults: ["compact_dex_converter_defaults"],
    shared_libs: [
        "libart",
        "libart-dexlayout",
        "libdexfile",
    ],
}

art_cc_binary {
    name: "compact_dex_converterd",
    defaults: [
        "art_debug_defaults",
        "compact_dex_converter_defaults",
    ],
    shared_libs: [
        "libartd",
        "libartd-dexlayout",
        "libdexfiled",
    ],
}

and change to:

art_cc_binary {
    name: "compact_dex_converter",
    defaults: ["compact_dex_converter_defaults"],
    shared_libs: [
        "libartbase",
        "libart",
        "libart-dexlayout",
        "libdexfile",
    ],
}

art_cc_binary {
    name: "compact_dex_converterd",
    defaults: [
        "art_debug_defaults",
        "compact_dex_converter_defaults",
    ],
    shared_libs: [
        "libartbased",
        "libartd",
        "libartd-dexlayout",
        "libdexfiled",
    ],
}

@superr

--- a/compact_dex_converter_main.cc	2018-09-11 09:14:21.928715800 +0300
+++ b/compact_dex_converter_main.cc	2018-09-03 23:56:40.856742000 +0300
@@ -36,7 +36,7 @@
 #include "base/os.h"
 #include "base/unix_file/fd_file.h"
 #include "dex/dex_file_loader.h"
-#include "mem_map.h"
+#include "base/mem_map.h"
 #include "runtime.h"
 
 namespace art {
@@ -67,19 +67,22 @@
 
   // We cannot use the ArtDexFileLoader since it only supports openning CompactDex
   // files from Vdex containers.
-  std::string error;
+  DexFileLoaderErrorCode error_code;
+  std::string error_msg;
   const bool kVerifyChecksum = false;
   const bool kVerify = true;  // Input file should always verify
   const DexFileLoader dex_file_loader;
   std::vector<std::unique_ptr<const DexFile>> dex_files;
   if (!dex_file_loader.OpenAll(reinterpret_cast<const uint8_t*>(content.data()),
                                content.size(),
                                file_name,
                                kVerify,
                                kVerifyChecksum,
-                               &error,
+                               &error_code,
+                               &error_msg,
                                &dex_files)) {
-    LOG(ERROR) << "Input Dex file open failed: " << error;
+    LOG(ERROR) << "Input Dex file open failed: " << error_msg;
     return 1;
   }
 
@@ -112,8 +115,8 @@
                                  dex_files[0].get(),
                                  0,
                                  &dex_container,
-                                 &error)) {
-    LOG(ERROR) << "Unable to process CompactDex file: " << error;
+                                 &error_msg)) {
+    LOG(ERROR) << "Unable to process CompactDex file: " << error_msg;
     return 1;
   }
 
@@ -132,9 +135,9 @@
                                       /*oat_dex_file*/ nullptr,
                                       /*verify*/ verify_output_file,
                                       /*verify_checksum*/ false,
-                                      &error);
+                                      &error_msg);
   if (new_dex_file == nullptr) {
-    LOG(ERROR) << "Unable to open dex file from memory: " << error;
+    LOG(ERROR) << "Unable to open dex file from memory: " << error_msg;
     return 1;
   }
 
@@ -251,3 +254,4 @@
 
   return art::DexlayoutDriver(argc, argv);
 }

@superr it seems that the libc++.so that I have compiled and shared is not compatible with your system due to some CPU missing features (most probably an old model not supporting all new instruction emitted from newer compiler version).

The reason that it works when running from within valgrind, is because valgrind places its own hooks to instrument allocations (including the one that contains the illegal instruction) and thus the original implementation is never invoked (SIGILL is not triggered).

Unfortunately I cannot do much from my side. However, you can try forking the necessary AOSP repos, apply the compact_dex_converter patch in the ART git repo and build yourself. Hopefully, when compiling libc++ the compiler will handle the older CPU issue.