quarkus: REST Client retry failed calls

Describe the bug

Using Quarkus 1.13.3.Final when I make the request fail on the other side by returning a Uni.createFrom().failure(new RuntimeException()); the client just hangs until the set amount of attempts with the preset delay happen but there is no sign of these attempts ever reaching the other service. The code at the remote endpoint only runs once.

Expected behavior

The client makes calls to the other service the number of times set in retry().atMost(NUMBER)

Actual behavior

The remote endpoint is only called once.

To Reproduce

See this stackoverflow thread for the details

REST Client:

@Path("/dataservice")
@RegisterRestClient(configKey = "restclient")
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public interface RestClientToDataService {

    @POST
    Uni<DataObject> getDataObject(String data);

}

REST Client properties:

restclient/mp-rest/url=http://localhost:${remote.service.port}

REST Client being injected and used on the caller side:

@Path("")
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public class ReactiveController {

    private final RestClientToDataService dataServiceClient;

    public ReactiveController(@RestClient RestClientToDataService dataServiceClient) {
        this.dataServiceClient = dataServiceClient;
    }


    @GET
    @Path("/test")
    public void test() {

        dataServiceClient.getDataObject("data")
                .onFailure().retry()
                .withBackOff(Duration.ofMillis(100), Duration.ofMillis(200))
                .atMost(6);
    }
}

The remote service side:

@Path("/dataservice")
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public class DataServiceController {

    @POST
    public Uni<DataObject> getDataObject(String data) {
        System.out.println(">>>>>>>>>>>>>>>> DATA CALL");
        return Uni.createFrom().failure(new RuntimeException("Failed DataServiceController for " + data));
    }
}

The caller service does not produce any logs but from the remote service logs it seems clear that the endpoint is either only called once despite setting the retry count to 6 or there might be some kind of caching in place for either side that I couldn’t find anything of in the documentation.

>>>>>>>>>>>>>>>> DATA CALL
2021-05-11 14:36:11,912 ERROR [org.jbo.res.rea.ser.cor.ExceptionMapping] (vert.x-eventloop-thread-21) Request failed : java.lang.RuntimeException: Failed DataServiceController for data
	at com.neticle.reactive.service.dataservice.DataServiceController.getDataObjects(DataServiceController.java:43)
	at com.neticle.reactive.service.dataservice.DataServiceController_Subclass.getDataObjects$$superaccessor1(DataServiceController_Subclass.zig:211)
	at com.neticle.reactive.service.dataservice.DataServiceController_Subclass$$function$$5.apply(DataServiceController_Subclass$$function$$5.zig:41)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:54)
	at io.quarkus.arc.runtime.devconsole.InvocationInterceptor.proceed(InvocationInterceptor.java:63)
	at io.quarkus.arc.runtime.devconsole.InvocationInterceptor.monitor(InvocationInterceptor.java:49)
	at io.quarkus.arc.runtime.devconsole.InvocationInterceptor_Bean.intercept(InvocationInterceptor_Bean.zig:521)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:41)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:41)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:32)
	at com.neticle.reactive.service.dataservice.DataServiceController_Subclass.getDataObjects(DataServiceController_Subclass.zig:166)
	at com.neticle.reactive.service.dataservice.DataServiceController$quarkusrestinvoker$getDataObjects_2fa034de4b478d9f7f54657d56e0c77c0a510d65.invoke(DataServiceController$quarkusrestinvoker$getDataObjects_2fa034de4b478d9f7f54657d56e0c77c0a510d65.zig:45)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:7)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:122)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

2021-05-11 14:36:11,913 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-21) HTTP Request to /dataservice?batch=1620736571899 failed, error id: 2c35e7ff-aeb6-48c1-b76f-0598a1b18294-34: java.lang.RuntimeException: Failed DataServiceController for data
	at com.neticle.reactive.service.dataservice.DataServiceController.getDataObjects(DataServiceController.java:43)
	at com.neticle.reactive.service.dataservice.DataServiceController_Subclass.getDataObjects$$superaccessor1(DataServiceController_Subclass.zig:211)
	at com.neticle.reactive.service.dataservice.DataServiceController_Subclass$$function$$5.apply(DataServiceController_Subclass$$function$$5.zig:41)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:54)
	at io.quarkus.arc.runtime.devconsole.InvocationInterceptor.proceed(InvocationInterceptor.java:63)
	at io.quarkus.arc.runtime.devconsole.InvocationInterceptor.monitor(InvocationInterceptor.java:49)
	at io.quarkus.arc.runtime.devconsole.InvocationInterceptor_Bean.intercept(InvocationInterceptor_Bean.zig:521)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:41)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:41)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:32)
	at com.neticle.reactive.service.dataservice.DataServiceController_Subclass.getDataObjects(DataServiceController_Subclass.zig:166)
	at com.neticle.reactive.service.dataservice.DataServiceController$quarkusrestinvoker$getDataObjects_2fa034de4b478d9f7f54657d56e0c77c0a510d65.invoke(DataServiceController$quarkusrestinvoker$getDataObjects_2fa034de4b478d9f7f54657d56e0c77c0a510d65.zig:45)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:7)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:122)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

2021-05-11 14:36:11,913 ERROR [org.jbo.res.rea.com.cor.AbstractResteasyReactiveContext] (vert.x-eventloop-thread-21) Request failed: java.lang.RuntimeException: Failed DataServiceController for data
	at com.neticle.reactive.service.dataservice.DataServiceController.getDataObjects(DataServiceController.java:43)
	at com.neticle.reactive.service.dataservice.DataServiceController_Subclass.getDataObjects$$superaccessor1(DataServiceController_Subclass.zig:211)
	at com.neticle.reactive.service.dataservice.DataServiceController_Subclass$$function$$5.apply(DataServiceController_Subclass$$function$$5.zig:41)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:54)
	at io.quarkus.arc.runtime.devconsole.InvocationInterceptor.proceed(InvocationInterceptor.java:63)
	at io.quarkus.arc.runtime.devconsole.InvocationInterceptor.monitor(InvocationInterceptor.java:49)
	at io.quarkus.arc.runtime.devconsole.InvocationInterceptor_Bean.intercept(InvocationInterceptor_Bean.zig:521)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:41)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:41)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:32)
	at com.neticle.reactive.service.dataservice.DataServiceController_Subclass.getDataObjects(DataServiceController_Subclass.zig:166)
	at com.neticle.reactive.service.dataservice.DataServiceController$quarkusrestinvoker$getDataObjects_2fa034de4b478d9f7f54657d56e0c77c0a510d65.invoke(DataServiceController$quarkusrestinvoker$getDataObjects_2fa034de4b478d9f7f54657d56e0c77c0a510d65.zig:45)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:7)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:122)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)