amoro: [Bug]: Optimizing process stuck after a few failures

What happened?

If an optimizing task fails constantly (e.g. because of corrupted files), the optimizing task would be stuck at the RUNNING state after a few retries, while the task has already failed at the optimizer side.

截屏2024-03-11 16 55 27

Affects Versions

0.5.1

What engines are you seeing the problem on?

AMS, Optimizer

How to reproduce

  1. create a table with some data that meets the optimizing conditions.
  2. truncate a data file that would take part in the optimizing compaction.
  3. AMS tries to optimize the table but fails and retries.
  4. after a few retries, AMS would lose track of the task, and leave the task stuck at RUNNING state.

Relevant log output

No response

Anything else

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

  • I agree to follow this project’s Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 35 (35 by maintainers)

Most upvoted comments

For me, this is why the state is not being updated.

"thrift-server-optimize-manager-20575" Id=646912 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@68e7d329 owned by "thrift-server-optimize-manager-20440" Id=641953
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@68e7d329
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at com.netease.arctic.server.persistence.StatedPersistentBase.invokeConsisitency(StatedPersistentBase.java:49)
    at com.netease.arctic.server.optimizing.TaskRuntime.tryCanceling(TaskRuntime.java:146)
    at com.netease.arctic.server.optimizing.OptimizingQueue$TableOptimizingProcess$$Lambda$1010/1409555188.accept(Unknown Source)
    at java.util.HashMap$Values.forEach(HashMap.java:981)
    at com.netease.arctic.server.optimizing.OptimizingQueue$TableOptimizingProcess.lambda$persistProcessCompleted$10(OptimizingQueue.java:604)
    at com.netease.arctic.server.optimizing.OptimizingQueue$TableOptimizingProcess$$Lambda$1007/808873318.run(Unknown Source)
    at com.netease.arctic.server.persistence.PersistentBase$$Lambda$278/886902042.accept(Unknown Source)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
    at com.netease.arctic.server.persistence.PersistentBase.doAsTransaction(PersistentBase.java:61)
    at com.netease.arctic.server.optimizing.OptimizingQueue.access$400(OptimizingQueue.java:71)
    at com.netease.arctic.server.optimizing.OptimizingQueue$TableOptimizingProcess.persistProcessCompleted(OptimizingQueue.java:603)
    at com.netease.arctic.server.optimizing.OptimizingQueue$TableOptimizingProcess.acceptResult(OptimizingQueue.java:437)
    at com.netease.arctic.server.optimizing.TaskRuntime.lambda$complete$0(TaskRuntime.java:103)
    at com.netease.arctic.server.optimizing.TaskRuntime$$Lambda$274/802047498.run(Unknown Source)
    at com.netease.arctic.server.persistence.PersistentBase$$Lambda$278/886902042.accept(Unknown Source)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
    at com.netease.arctic.server.persistence.PersistentBase.doAsTransaction(PersistentBase.java:61)
    at com.netease.arctic.server.persistence.StatedPersistentBase.invokeConsisitency(StatedPersistentBase.java:52)
    at com.netease.arctic.server.optimizing.TaskRuntime.complete(TaskRuntime.java:76)
    at com.netease.arctic.server.DefaultOptimizingService.completeTask(DefaultOptimizingService.java:230)
    at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.netease.arctic.server.utils.ThriftServiceProxy.invoke(ThriftServiceProxy.java:56)
    at com.sun.proxy.$Proxy48.completeTask(Unknown Source)
    at com.netease.arctic.ams.api.OptimizingService$Processor$completeTask.getResult(OptimizingService.java:583)
    at com.netease.arctic.ams.api.OptimizingService$Processor$completeTask.getResult(OptimizingService.java:562)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
    at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:138)
    at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:524)
    at org.apache.thrift.server.Invocation.run(Invocation.java:18)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"thrift-server-optimize-manager-20440" Id=641953 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@77281271 owned by "thrift-server-optimize-manager-20575" Id=646912
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@77281271
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at com.netease.arctic.server.optimizing.OptimizingQueue$TableOptimizingProcess.acceptResult(OptimizingQueue.java:407)
    at com.netease.arctic.server.optimizing.TaskRuntime.lambda$complete$0(TaskRuntime.java:103)
    at com.netease.arctic.server.optimizing.TaskRuntime$$Lambda$274/802047498.run(Unknown Source)
    at com.netease.arctic.server.persistence.PersistentBase$$Lambda$278/886902042.accept(Unknown Source)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
    at com.netease.arctic.server.persistence.PersistentBase.doAsTransaction(PersistentBase.java:61)
    at com.netease.arctic.server.persistence.StatedPersistentBase.invokeConsisitency(StatedPersistentBase.java:52)
    at com.netease.arctic.server.optimizing.TaskRuntime.complete(TaskRuntime.java:76)
    at com.netease.arctic.server.DefaultOptimizingService.completeTask(DefaultOptimizingService.java:230)
    at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.netease.arctic.server.utils.ThriftServiceProxy.invoke(ThriftServiceProxy.java:56)
    at com.sun.proxy.$Proxy48.completeTask(Unknown Source)
    at com.netease.arctic.ams.api.OptimizingService$Processor$completeTask.getResult(OptimizingService.java:583)
    at com.netease.arctic.ams.api.OptimizingService$Processor$completeTask.getResult(OptimizingService.java:562)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
    at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:138)
    at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:524)
    at org.apache.thrift.server.Invocation.run(Invocation.java:18)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)