quarkus: zip file closed errors

Describe the bug

Hello, for a little while now our quarkus apps have been getting frequent zip file closed errors like this one. This happens randomly.

ERROR [com.xxxx.ThrowableMapper] (vert.x-eventloop-thread-17) POST http://xxx failed: zip file closed: java.lang.IllegalStateException: zip file closed
at java.base/java.util.zip.ZipFile.ensureOpen(ZipFile.java:831)
at java.base/java.util.zip.ZipFile.getEntry(ZipFile.java:330)
at java.base/java.util.jar.JarFile.getEntry(JarFile.java:518)
at io.quarkus.bootstrap.runner.JarResource.getResourceData(JarResource.java:74)
at io.quarkus.bootstrap.runner.RunnerClassLoader.loadClass(RunnerClassLoader.java:99)
at io.quarkus.bootstrap.runner.RunnerClassLoader.loadClass(RunnerClassLoader.java:65)
at com.xxx.XXXResource_Bean.proxy(Unknown Source)
at com.xxx.XXXtResource_Bean.get(Unknown Source)
xxxx
at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:471)
at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:484)
at io.quarkus.arc.impl.ArcContainerImpl$1.get(ArcContainerImpl.java:288)
at io.quarkus.arc.impl.ArcContainerImpl$1.get(ArcContainerImpl.java:285)
at io.quarkus.arc.runtime.BeanContainerImpl$1.create(BeanContainerImpl.java:36)
at io.quarkus.resteasy.reactive.common.runtime.ArcBeanFactory.createInstance(ArcBeanFactory.java:27)
at org.jboss.resteasy.reactive.server.handlers.InstanceHandler.handle(InstanceHandler.java:26)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:113)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:142)
at org.jboss.resteasy.reactive.server.vertx.VertxResteasyReactiveRequestContext$1$1.handle(VertxResteasyReactiveRequestContext.java:79)
at org.jboss.resteasy.reactive.server.vertx.VertxResteasyReactiveRequestContext$1$1.handle(VertxResteasyReactiveRequestContext.java:76)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:246)
at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:43)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)

At different occasions it occurs for different XXXResource_Bean classes, but the upper part of the stacktrace is always the same.

I Realize this is a very generic error but currently we have no clue how to track that, unfortunately it’s not telling me which zip file it is (I assume a jar file).

Has anyone ever seen a similar behavior? Any hints welcome. We already switched machines a few times since we suspected faulty drives.

This happens in production and our staging environment. The endpoints failing are http requests doing basic CRUD via vertx reactive pg client. Sometimes its a HTTP resource bean, sometimes an intermediate service bean thats part of the stacktrace

Expected behavior

When calling an endpoint of the app it should reliably answer

Actual behavior

At times there is multitude of zip file errors. Only a restart fixes that.

How to Reproduce?

Not explicitly reproducible

Output of uname -a or ver

Linux stage-myad2 5.10.0-21-cloud-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

Output of java -version

openjdk version “17.0.7”

GraalVM version (if different from Java)

openjdk version “17.0.7”

Quarkus version or git rev

Quarkus 2.13.7 (production mode)

Build tool (ie. output of mvnw --version or gradlew --version)

Gradle 7.5.1

Additional information

There is a mention of possible race in JarResource.java on line 154 https://github.com/quarkusio/quarkus/blob/88c2a7b5e7638509e887c5bc0a39ed9840a537ce/independent-projects/bootstrap/runner/src/main/java/io/quarkus/bootstrap/runner/JarResource.java#L154 Maybe this is related, but this is quite old code, I’m wondering why we only ‘recently’ are having issues with this kind of error. We reported this in quarkusio zulipchat as well https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/zip.20file.20closed