arrow: [Java][Dataset] JNI Error when reading parquet file
Describe the bug, including details regarding any error messages, version, and platform.
An JNI error happened when I try to read a parquet file with arrow-dataset (15.0.0) in a java program
Runtime : OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)
It seems that’s due to the usage of NativeMemoryPool.createListenable( DirectReservationListener.instance())
instead of NativeMemoryPool.getDefault()
By the way the error not happened with all parquet files
Full code for reproduce the error
https://gist.github.com/fb64/71880cde297bc5234b02b68b785670fd
Stack trace
/Users/runner/work/crossbow/crossbow/arrow/java/dataset/src/main/cpp/jni_util.cc:79: Failed to update reservation while freeing bytes: JNIEnv was not attached to current thread
0 jnilib-10391875251891153172.tmp 0x00000001424a99ac _ZN5arrow4util7CerrLogD2Ev + 204
1 jnilib-10391875251891153172.tmp 0x00000001424a98ce _ZN5arrow4util7CerrLogD0Ev + 14
2 jnilib-10391875251891153172.tmp 0x00000001424a1112 _ZN5arrow4util8ArrowLogD1Ev + 34
3 jnilib-10391875251891153172.tmp 0x00000001412608ed _ZN5arrow7dataset3jni31ReservationListenableMemoryPool4Impl4FreeEPhxx + 237
4 jnilib-10391875251891153172.tmp 0x00000001427b2acb _ZN5arrow10PoolBufferD2Ev + 59
5 jnilib-10391875251891153172.tmp 0x00000001427b268e _ZN5arrow10PoolBufferD0Ev + 14
6 jnilib-10391875251891153172.tmp 0x000000014140f508 _ZN7parquet14SerializedFileD2Ev + 168
7 jnilib-10391875251891153172.tmp 0x000000014140ebfe _ZN7parquet14SerializedFileD0Ev + 14
8 jnilib-10391875251891153172.tmp 0x000000014153a8ef _ZN7parquet5arrow12_GLOBAL__N_114FileReaderImplD0Ev + 159
9 jnilib-10391875251891153172.tmp 0x000000014155048a _ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureINS0_5EmptyEE20WrapResultOnComplete8CallbackINSA_14ThenOnCompleteIZN7parquet5arrow17RowGroupGenerator9FetchNextEvEUlvE_NSA_17PassthruOnFailureISH_EEEEEEED0Ev + 122
10 jnilib-10391875251891153172.tmp 0x00000001424a503d _ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE + 189
11 jnilib-10391875251891153172.tmp 0x00000001428471d1 _ZN5arrow6FutureINS_8internal5EmptyEE14DoMarkFinishedENS_6ResultIS2_EE + 129
12 jnilib-10391875251891153172.tmp 0x0000000142847055 _ZN5arrow6FutureINS_8internal5EmptyEE12MarkFinishedIS2_vEEvNS_6StatusE + 53
13 jnilib-10391875251891153172.tmp 0x000000014154f4a4 _ZZN5arrow8internal8Executor10DoTransferINS0_5EmptyENS_6FutureIS3_EENS_6StatusEEENS4_IT_EES8_bENUlRKS6_E_clESA_ + 100
14 jnilib-10391875251891153172.tmp 0x00000001424a5650 _ZN5arrow8internal6FnOnceIFvvEE6FnImplIZNS_18ConcreteFutureImpl21RunOrScheduleCallbackERKNSt3__110shared_ptrINS_10FutureImplEEEONS8_14CallbackRecordEbEUlvE_E6invokeEv + 32
15 jnilib-10391875251891153172.tmp 0x000000014248e055 _ZNSt3__1L14__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN5arrow8internal10ThreadPool21LaunchWorkersUnlockedEiE4$_10EEEEEPvSC_ + 709
16 libsystem_pthread.dylib 0x00007ff810c77202 _pthread_start + 99
17 libsystem_pthread.dylib 0x00007ff810c72bab thread_start + 15
Component(s)
Java
About this issue
- Original URL
- State: closed
- Created 5 months ago
- Reactions: 1
- Comments: 23 (20 by maintainers)
Commits related to this issue
- GH-39919: [C++][Dataset] Add missing Protobuf static link dependency We need to link system libprotobuf.a too when we link libarrow.a, ORC is enabled and system Protobuf is used. — committed to kou/arrow by kou 5 months ago
- GH-39919: [C++][Dataset] Add missing Protobuf static link dependency (#40015) ### Rationale for this change We need to link system libprotobuf.a too when we link libarrow.a, ORC is enabled and syste... — committed to apache/arrow by kou 5 months ago
- GH-39919: [C++][Dataset] Add missing Protobuf static link dependency (#40015) ### Rationale for this change We need to link system libprotobuf.a too when we link libarrow.a, ORC is enabled and syste... — committed to dgreiss/arrow by kou 5 months ago
- GH-39919: [C++][Dataset] Add missing Protobuf static link dependency (#40015) ### Rationale for this change We need to link system libprotobuf.a too when we link libarrow.a, ORC is enabled and syste... — committed to zanmato1984/arrow by kou 5 months ago
- GH-39919: [C++][Dataset] Add missing Protobuf static link dependency (#40015) ### Rationale for this change We need to link system libprotobuf.a too when we link libarrow.a, ORC is enabled and syste... — committed to thisisnic/arrow by kou 5 months ago
- GH-39919: [C++][Dataset] Add missing Protobuf static link dependency (#40015) ### Rationale for this change We need to link system libprotobuf.a too when we link libarrow.a, ORC is enabled and syste... — committed to apache/arrow by kou 5 months ago
Can we use #20379 for further discussion for this problem?
Thanks for checking the result. I’ll merge #40015 tomorrow if nobody objects it.
Hi @fb64, I’m sorry to discuss other errors related to this issue. Please let me review the error
...JNIEnv was not attached to the current thread...
to see what is going on.Any hint about the main error of this issue ?
/Users/runner/work/crossbow/crossbow/arrow/java/dataset/src/main/cpp/jni_util.cc:79: Failed to update reservation while freeing bytes: JNIEnv was not attached to current thread
It also happened with Arrow 14 BTWIt’s not excatly the same issue, we also faced to it on Kotlin dataframe : https://github.com/Kotlin/dataframe/pull/577#issuecomment-1918952628 It seems that it only happened on Linux, I think it’s native library compilation issue as the library seems using Protobof but is not statically linked to it. It’s working with arrow 14 by the way …