accumulo: Compaction-Planner worker thread dies during compaction
Describe the bug Compaction planner threads are dying with an IllegalArgumentException during an internal compaction.
[threads.AccumuloUncaughtExceptionHandler] ERROR: Caught an Exception in Thread[CompactionPlanner-Worker-35,5,main]. Thread is dead.
java.lang.IllegalArgumentException: Unknown runnable type io.opentelemetry.context.Context$$Lambda$243/0x00000008403f8040
at org.apache.accumulo.tserver.compactions.InternalCompactionExecutor.getJob(InternalCompactionExecutor.java:152) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.util.Comparator.lambda$comparing$ea9a8b3a$1(Comparator.java:436) ~[?:?]
at java.util.concurrent.PriorityBlockingQueue.siftUpUsingComparator(PriorityBlockingQueue.java:377) ~[?:?]
at java.util.concurrent.PriorityBlockingQueue.offer(PriorityBlockingQueue.java:488) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1347) ~[?:?]
at org.apache.accumulo.core.util.threads.ThreadPools$1.execute(ThreadPools.java:186) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.compactions.InternalCompactionExecutor.submit(InternalCompactionExecutor.java:180) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.compactions.CompactionService.submitCompactionJob(CompactionService.java:369) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.compactions.CompactionService.lambda$submitCompaction$4(CompactionService.java:247) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at io.opentelemetry.context.Context.lambda$wrap$1(Context.java:207) ~[opentelemetry-context-1.7.1.jar:1.7.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.opentelemetry.context.Context.lambda$wrap$1(Context.java:207) ~[opentelemetry-context-1.7.1.jar:1.7.1]
at java.lang.Thread.run(Thread.java:829) [?:?]
Versions :
- Affected version: Current head of main (f8bb900ae080fe0f54dfe04f9e1ad8c4dd2e7930 as of writing this). Though I speculate issue may be due to changes in b6a466348807fe0d23203f3776cbc3119d5a037d.
To Reproduce Steps to reproduce the behavior
- Setup an Accumulo instance
- Create a table utilizing the new compaction server props (I used accumulo-testing for this)
- Use accumulo-testing to continuously-ingest data
- After a short time, the exception gets thrown. (Seems a manual compaction will throw an exception as well)
Additional context The error message seems to point to it being related to the recent open-telemetry addition in #2259.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 21 (21 by maintainers)
Commits related to this issue
- Improve generics in InternalCompactionExecutor Fix #2362 by using InternalJob as the generic type instead of Runnable for the PriorityBlockingQueue and its corresponding Comparator in InternalCompact... — committed to ctubbsii/accumulo by ctubbsii 3 years ago
- Wrap and unwrap tasks in thread pools properly Wrap Runnable and Callable objects properly inside thread pools in a way that allows us to unwrap them to make comparisons in priority blocking queues, ... — committed to ctubbsii/accumulo by ctubbsii 2 years ago
- Wrap and unwrap tasks in thread pools properly (#2442) Wrap Runnable and Callable objects properly inside thread pools in a way that allows us to unwrap them to make comparisons in priority blocking... — committed to apache/accumulo by ctubbsii 2 years ago
The server-side logs are in the details portion of the issue body. I will see if I can unearth any other helpful logs. I will try to reproduce it manually but if that cannot be done then its possible the issue resides in how we set the properties in accumulo-testing.
After some thought, I realized my first attempt won’t work, but I think I can still fix this.