smallrye-fault-tolerance: Semaphore behavior in Bulkhead strategy (add queueSize and timeout)

With JDK21 we have Virtual Threads: they are cheap, and you can block them without any thoughts

Could you add an option to Semaphore based Bulkhead strategies (e.g. timeout: long millis), which, when enabled, changes the behavior of Semaphore from tryAcquire() to tryAcquire(long timeout, TimeUnit unit/*MILLISECONDS*/)

So, if one runs a code with Bulkhead and sets limit to 10, then only 10 threads would run concurrently and other (millions) would be put into “waiting queue”

The application code itself would be much simpler.

Now I have to use plain Semaphore in addition to FT 😦

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 32 (32 by maintainers)

Most upvoted comments

@Ladicek Please! 🙏😥 As you see no timeouts 🤝 (I have managed to emulate them) But I need two semaphores very badly… https://github.com/smallrye/smallrye-fault-tolerance/pull/1015 → this is your own io.smallrye.faulttolerance.core.bulkhead.FutureThreadPoolBulkhead, but without Future and used in smallrye-fault-tolerance-standalone

It is optional, for those who knows, everything works as before by default.

🙏

OK, so in #989, I added withThreadOffloadExecutor() to specify an executor explicitly, in case the default one (provided by StandaloneFaultTolerance.configure()) is not the right one.

I also realized that FaultTolerance.call() is not the only method you can use. It you want to guard a Supplier, you can call get(), and if you want to guard a Runnable like in this example, you can call run(). With a Runnable, you don’t need to produce a meaningless return value.

Combining both of the above, you can write:

@Test
public void test() throws Exception {
    var ft = FaultTolerance.<Integer>createAsync()
            .withDescription("_bulkhead")
            .withBulkhead().limit(2).queueSize(100).done()
            .withThreadOffload(true)
            .withThreadOffloadExecutor(executor)
            .build();

    var total = new AtomicInteger();
    var concurrent = new AtomicInteger();
    var endTest = new CountDownLatch(20);

    for (int i = 0; i < 20; i++) {
        ft.run(() -> {
            total.incrementAndGet();
            concurrent.incrementAndGet();
            try {
                Thread.sleep(20);
            } catch (Throwable ignore) {
            }
            System.out.printf("Data sent: total=%s, concurrent=%s%n", total, concurrent);
            concurrent.decrementAndGet();
            endTest.countDown();
        });
    }

    endTest.await();
}

Ok, if we call it time-in-the-queue timeout, it makes things complicated.

So it is not 😃 ⇒ It is workSemaphore-acquire-timeout. It has sense even in case of CF version. If you have spikes in the traffic, then it is better to have some timeout to acquire it, don’t fail immediately.

It helps to smooth out the peaks, keep order of messages, prevent lost of packets, etc

Without timeout, FT Bulkhead simply fails the task.

Ah ah ah, you’re right, my bad. I didn’t look properly at how the Semaphore methods differ.

New times, new rules maybe? With VT you can delegate queuing to them. It is an official position of JDK architects.

SmallRye Fault Tolerance has to work in the actual real world, which will take years to fully adopt virtual threads. Sorry, but this is just the sad state of reality.

Actually one more thing, I probably didn’t express that clearly above. The usage of CompletionStage in SmallRye Fault Tolerance doesn’t mean you’re required to use non-blocking concurrency. If you do thread offloads (either manually, or ask SmallRye Fault Tolerance to do it for you), you can use blocking concurrency just fine. In that case, you will only create already complete CompletionStages (CompletableFuture.completedStage() and failedStage()). If you want to block on the CompletionStage returned by SmallRye Fault Tolerance, you can call toCompletableFuture(), and that’s guaranteed to work. This indeed entails certain amount of code overhead, but not very big I believe.

Ah I see. If you have 20 threads competing for a semaphore that only hands out 2 permits, you indeed have an implicit form of queueing, but at the cost of having 20 threads. That may be fine with virtual threads, but it definitely is not fine with platform threads.

Now, when it comes to

limit + queue ≈ tryAcquire with timeout

that’s not true. You don’t need the timeout there, that doesn’t really change anything when it comes to the queueing semantics. It is an extra cherry on top.

In SmallRye Fault Tolerance, bulkheads implement an explicit form of queueing. Tasks are put into an actual queue and are not submitted to an executor until they obtain a permit. (Aside: in the annotation-based API, this is only true for CompletionStage-returning tasks and not for Future-returning tasks, but the programmatic API doesn’t support Future-returning tasks at all.)

Ah OK, so you want the queued tasks to have a timeout? And if the task is in the queue for too long, it should fail immediately? That seems like a reasonable request.

Don’t let anyone tell you that virtual threads are some form of a panacea; they are still just threads. The only thing that might possibly change in the future is that we will no longer need to deal with non-blocking concurrency (aka event loops or “reactive”); blocking concurrency (aka multi-threading) will be enough. But we’re certainly not in that world yet; it will take years for the ecosystem to adapt.

Some people equate the word “asynchronous” with one or the other form of concurrency I mentioned above. In SmallRye Fault Tolerance, I use the word “asynchronous” in a more abstract manner, close to the dictionary definition of “not happening at the same time”.

If an operation is asynchronous, it means its result is not available right here right now; the idea of reifying that concept into a future is decades old. As it turns out, if we restrict ourselves to the JDK, the best type for representing futures is currently CompletionStage / CompletableFuture, because it supports both styles of concurrency. So that’s what SmallRye Fault Tolerance uses. Maybe at some point, this won’t be necessary, but as I mentioned above, we’re years away from that.

Now, here’s a piece of code that uses bulkhead with a queue (which requires the operation to be asynchronous, otherwise there’s no way to implement queueing):

@Test
public void test() throws Exception {
    var ft = FaultTolerance.<Integer>createAsync()
            .withDescription("_bulkhead")
            .withBulkhead().limit(2).queueSize(100).done()
            .build();

    var total = new AtomicInteger();
    var concurrent = new AtomicInteger();
    var endTest = new CountDownLatch(20);

    for (int i = 0; i < 20; i++) {
        Runnable task = () -> {
            try {
                total.incrementAndGet();
                ft.call(() -> {
                    concurrent.incrementAndGet();
                    try {
                        Thread.sleep(20);
                    } catch (Throwable ignore) {
                    }
                    System.out.printf("Data sent: total=%s, concurrent=%s%n", total, concurrent);
                    concurrent.decrementAndGet();
                    return null;
                });
            } catch (Exception e) {
                System.out.printf("FT.call failed: total=%s%n", total);
            } finally {
                endTest.countDown();
            }
        };

        executor.execute(task);
    }

    endTest.await();
}

Apparently, returning null works in this particular case, but I wouldn’t rely on it for anything real. I really would return CompletableFuture.completedStage(null);.

If I rely on SmallRye Fault Tolerance to submit the tasks to an executor on its own, instead of doing it manually, that code becomes:

@Test
public void test() throws Exception {
    var ft = FaultTolerance.<Integer>createAsync()
            .withDescription("_bulkhead")
            .withBulkhead().limit(2).queueSize(100).done()
            .withThreadOffload(true)
            .build();

    var total = new AtomicInteger();
    var concurrent = new AtomicInteger();
    var endTest = new CountDownLatch(20);

    for (int i = 0; i < 20; i++) {
        ft.call(() -> {
            total.incrementAndGet();
            concurrent.incrementAndGet();
            try {
                Thread.sleep(20);
            } catch (Throwable ignore) {
            }
            System.out.printf("Data sent: total=%s, concurrent=%s%n", total, concurrent);
            concurrent.decrementAndGet();
            endTest.countDown();
            return CompletableFuture.completedStage(null);
        });
    }

    endTest.await();
}

This works perfectly well.

You want something like

  var ft = FaultTolerance.<Integer>createAsync()
    .withDescription("_bulkhead")
    .withBulkhead().limit(2).queueSize(100).done()
    .build();

Then, your task needs to return CompletionStage, which shouldn’t be an issue, you’d just replace return null; with return CompletableFuture.completedStage(null);.

If all tasks in the application are submitted to the same executor, you could even add withThreadOffload(true) to the FaultTolerance builder and set that executor using StandaloneFaultTolerance.configure(). That way, you wouldn’t even have to create the Runnable and submit it to the executor manually, SmallRye Fault Tolerance would do that for you. (Actually, this is quite a limitation. At least in the programmatic API, we should be able to specify the executor for each FaultTolerance object separately. Let me file an issue for that.)