open-liberty: InstantOn beta2 - crash in JVM when create checkpoint on a AMD Linux and restore on a different Linux
Using InstantOn beta2 image with Java17
********************************************************************************
product = Open Liberty 22.0.0.13-beta (wlp-1.0.71.cl221220221107-1900)
wlp.install.dir = /opt/ol/wlp/
server.output.dir = /opt/ol/wlp/output/defaultServer/
java.home = /opt/java/openjdk
java.version = 17.0.5-ea
java.runtime = IBM Semeru Runtime Open Edition (17.0.5-ea+8)
os = Linux (5.14.0-70.26.1.el9_0.x86_64; amd64) (en_US)
process = 126@99ca81e12bd2
Classpath = /opt/ol/wlp/bin/tools/ws-server.jar:/opt/ol/wlp/bin/tools/ws-javaagent.jar
Java Library path = /opt/java/openjdk/lib/default:/opt/java/openjdk/lib:/usr/lib64:/usr/lib
********************************************************************************
- Create checkpoints on AMD Ubuntu 22.04, restore the image on Intel RHEL9, it fails with the following errors in restore.log
# cat checkpoint/restore.log
Warn (criu/kerndat.c:1103): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file
Error (criu/libnetlink.c:84): Can't send request message: Permission denied
Error (criu/libnetlink.c:84): Can't send request message: Permission denied
Error (criu/libnetlink.c:84): Can't send request message: Permission denied
Error (criu/libnetlink.c:84): Can't send request message: Permission denied
Error (criu/libnetlink.c:84): Can't send request message: Permission denied
Error (criu/libnetlink.c:84): Can't send request message: Permission denied
Warn (criu/kerndat.c:1103): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file
122: Error (criu/sockets.c:461): sockets: Can't set 1:72 (len 4): Protocol not available
122: Error (criu/files.c:1213): Unable to open fd=14 id=0x48
Error (criu/cr-restore.c:1498): 122 exited, status=1
Error (criu/cr-restore.c:2531): Restoring FAILED.
- Create checkpoint on AMD RHEL 9 and restore the image on Intel RHEL 9, the restore also fails, but it has more information in the log
# podman run --cap-add=CHECKPOINT_RESTORE --cap-add=NET_ADMIN --cap-add=SYS_PTRACE --security-opt seccomp=compprofile.json -v /proc/sys/kernel/ns_last_pid:/proc/sys/kernel/ns_last_pid --detach -p 9080:9080 -p 9443:9443 --name microwebapp-restore-app-amd-rhel9 microwebee9-ckpt-app:amd-rhel9
3949f75bc98d45b823d19c3e607a9462dd2f99c6c9466940670bf23ea880291a
# podman logs microwebapp-restore-app-amd-rhel9
[AUDIT ] CWWKZ0001I: Application microwebapp started in 0.133 seconds.
Unhandled exception
Type=Illegal instruction vmState=0x00040000
J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=00000002
Handler1=00007FEE7DAFC3F0 Handler2=00007FEE7F0D4A80
RDI=00007FEE24007350 RSI=00007FEE2400736C RAX=2800000000000000 RBX=00007FEE24007350
RCX=0000000000000020 RDX=0000000000000000 R8=00007FEE240073AC R9=00000000078BFFFF
R10=00000000209C01AB R11=00007FEE5C0089E0 R12=00007FEE2400736C R13=00000000FF6F53D8
R14=0000000000B37F98 R15=00007FEE353FC788
RIP=00007FEE3D42AE04 GS=0000 FS=0000 RSP=00007FEE353FC3E8
EFlags=0000000000010246 CS=0033 RBP=00000000FF6F53D8 ERR=0000000000000000
TRAPNO=0000000000000006 OLDMASK=0000000000000000 CR2=0000000000000000
xmm0 98badcfe10325476 (f: 271733888.000000, d: -1.507306e-189)
xmm1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm2 00007fee7dcaa330 (f: 2110432000.000000, d: 6.949640e-310)
xmm3 08090a0b0c0d0e0f (f: 202182160.000000, d: 5.924543e-270)
xmm4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm7 0000000000000028 (f: 40.000000, d: 1.976263e-322)
xmm8 98badcfe10325476 (f: 271733888.000000, d: -1.507306e-189)
xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm10 000000ff00000000 (f: 0.000000, d: 5.411089e-312)
xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/lib64/libcrypto.so.1.1
Module_base_address=00007FEE3D25B000
Target=2_90_20221018_61 (Linux 5.14.0-70.26.1.el9_0.x86_64)
CPU=amd64 (4 logical CPUs) (0x1e04a0000 RAM)
----------- Stack Backtrace -----------
(0x00007FEE3D42AE04 [libcrypto.so.1.1+0x1cfe04])
SHA1_Final+0x70 (0x00007FEE3D42E950 [libcrypto.so.1.1+0x1d3950])
EVP_DigestFinal_ex+0x28 (0x00007FEE3D3B9808 [libcrypto.so.1.1+0x15e808])
Java_jdk_crypto_jniprovider_NativeCrypto_DigestComputeAndReset+0xc5 (0x00007FEE5C008AA5 [libjncrypto.so+0x3aa5])
ffi_call_unix64+0x52 (0x00007FEE7DC7D59A [libj9vm29.so+0x1e659a])
ffi_call_int+0x1a2 (0x00007FEE7DC7C732 [libj9vm29.so+0x1e5732])
_ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0x142b8 (0x00007FEE7DB68878 [libj9vm29.so+0xd1878])
bytecodeLoopCompressed+0xad (0x00007FEE7DB545AD [libj9vm29.so+0xbd5ad])
(0x00007FEE7DC0B852 [libj9vm29.so+0x174852])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2022/11/15 00:48:13 - please wait.
JVMDUMP032I JVM requested System dump using '/opt/ol/wlp/output/defaultServer/core.20221115.004813.126.0001.dmp' in response to an event
[AUDIT ] CWWKC0452I: The Liberty server process resumed operation from a checkpoint in 0.214 seconds.
[AUDIT ] CWWKF0012I: The server installed the following features: [appSecurity-4.0, cdi-3.0, checkpoint-1.0, concurrent-2.0, distributedMap-1.0, expressionLanguage-4.0, jndi-1.0, json-1.0, jsonb-2.0, jsonp-2.0, jwt-1.0, microProfile-5.0, monitor-1.0, mpConfig-3.0, mpFaultTolerance-4.0, mpHealth-4.0, mpJwt-2.0, mpMetrics-4.0, mpOpenAPI-3.0, mpOpenTracing-3.0, mpRestClient-3.0, pages-3.0, restfulWS-3.0, restfulWSClient-3.0, servlet-5.0, ssl-1.0, transportSecurity-1.0].
[AUDIT ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 0.267 seconds.
Unhandled exception
Type=Illegal instruction vmState=0x00040000
J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=00000002
Handler1=00007FEE7DAFC3F0 Handler2=00007FEE7F0D4A80
RDI=00007FEDE8020BF0 RSI=00000000FF9BF148 RAX=0000000000000000 RBX=00007FEDE8020BF0
RCX=00007FEE3D437D00 RDX=0000000000000001 R8=00000000006FA000 R9=00000000078BFFFF
R10=00000000FFF83203 R11=00000000209C01AB R12=00000000FF9BF148 R13=0000000000700DD8
R14=0000000000000040 R15=00007FEDE80074F0
RIP=00007FEE3D437FAB GS=0000 FS=0000 RSP=00007FEE3C749408
EFlags=0000000000010206 CS=0033 RBP=0000000000000040 ERR=0000000000000000
TRAPNO=0000000000000006 OLDMASK=0000000000000000 CR2=0000000000000000
xmm0 b3b69710b6cc9616 (f: 3066861056.000000, d: -1.405785e-59)
xmm1 510e527f9b05688c (f: 2600823040.000000, d: 2.876275e+82)
xmm2 1f83d9ab5be0cd19 (f: 1541459200.000000, d: 7.229008e-157)
xmm3 427f527f7442667e (f: 1950508672.000000, d: 2.152449e+12)
xmm4 72587944601d5f05 (f: 1612537600.000000, d: 6.527639e+242)
xmm5 0445415f6f620263 (f: 1868694144.000000, d: 4.362171e-288)
xmm6 3636363636363636 (f: 909522496.000000, d: 1.519791e-47)
xmm7 0405060700010203 (f: 66051.000000, d: 2.696622e-289)
xmm8 0405060700010203 (f: 66051.000000, d: 2.696622e-289)
xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm10 1f83d9ab5be0cd19 (f: 1541459200.000000, d: 7.229008e-157)
xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/lib64/libcrypto.so.1.1
Module_base_address=00007FEE3D25B000
Target=2_90_20221018_61 (Linux 5.14.0-70.26.1.el9_0.x86_64)
CPU=amd64 (4 logical CPUs) (0x1e04a0000 RAM)
----------- Stack Backtrace -----------
(0x00007FEE3D437FAB [libcrypto.so.1.1+0x1dcfab])
SHA256_Update+0x12d (0x00007FEE3D43B69D [libcrypto.so.1.1+0x1e069d])
Java_jdk_crypto_jniprovider_NativeCrypto_DigestUpdate+0x58 (0x00007FEE5C008988 [libjncrypto.so+0x3988])
ffi_call_unix64+0x52 (0x00007FEE7DC7D59A [libj9vm29.so+0x1e659a])
ffi_call_int+0x1a2 (0x00007FEE7DC7C732 [libj9vm29.so+0x1e5732])
_ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0x142b8 (0x00007FEE7DB68878 [libj9vm29.so+0xd1878])
bytecodeLoopCompressed+0xad (0x00007FEE7DB545AD [libj9vm29.so+0xbd5ad])
(0x00007FEE7DC0B852 [libj9vm29.so+0x174852])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2022/11/15 00:48:13 - please wait.
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" specifies that the core dump is to be piped to an external program. Attempting to rename either core or core.39.
JVMDUMP012E Error in System dump: The core file created by child process with pid = 39 was not found. Expected to find core file with name "/opt/ol/wlp/output/defaultServer/core.39"
JVMDUMP027W The requested heapdump has not been produced because another component is holding the VM exclusive lock.
JVMDUMP032I JVM requested Java dump using '/opt/ol/wlp/output/defaultServer/javacore.20221115.004813.126.0004.txt' in response to an event
JVMDUMP032I JVM requested System dump using '/opt/ol/wlp/output/defaultServer/core.20221115.004813.126.0002.dmp' in response to an event
JVMDUMP010I Java dump written to /opt/ol/wlp/output/defaultServer/javacore.20221115.004813.126.0004.txt
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" specifies that the core dump is to be piped to an external program. Attempting to rename either core or core.46.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (15 by maintainers)
@tajila yes I have the core files corefiles.zip
Yes, the issue occur when both vms are AMD. The following is restore.log on AMD RHEL