smallrye-fault-tolerance: Semaphore behavior in Bulkhead strategy (add queueSize and timeout)
With JDK21 we have Virtual Threads: they are cheap, and you can block them without any thoughts
Could you add an option to Semaphore based Bulkhead strategies (e.g. timeout: long millis),
which, when enabled, changes the behavior of Semaphore from
tryAcquire()
to
tryAcquire(long timeout, TimeUnit unit/*MILLISECONDS*/)
So, if one runs a code with Bulkhead and sets limit to 10, then only 10 threads would run concurrently and other (millions) would be put into “waiting queue”
The application code itself would be much simpler.
Now I have to use plain Semaphore in addition to FT 😦
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 32 (32 by maintainers)
@Ladicek Please! 🙏😥 As you see no timeouts 🤝 (I have managed to emulate them) But I need two semaphores very badly… https://github.com/smallrye/smallrye-fault-tolerance/pull/1015 → this is your own
io.smallrye.faulttolerance.core.bulkhead.FutureThreadPoolBulkhead, but without Future and used in smallrye-fault-tolerance-standaloneIt is optional, for those who knows, everything works as before by default.
🙏
OK, so in #989, I added
withThreadOffloadExecutor()to specify an executor explicitly, in case the default one (provided byStandaloneFaultTolerance.configure()) is not the right one.I also realized that
FaultTolerance.call()is not the only method you can use. It you want to guard aSupplier, you can callget(), and if you want to guard aRunnablelike in this example, you can callrun(). With aRunnable, you don’t need to produce a meaningless return value.Combining both of the above, you can write:
Ok, if we call it time-in-the-queue timeout, it makes things complicated.
So it is not 😃 ⇒ It is workSemaphore-acquire-timeout. It has sense even in case of CF version. If you have spikes in the traffic, then it is better to have some timeout to acquire it, don’t fail immediately.
It helps to smooth out the peaks, keep order of messages, prevent lost of packets, etc
Ah ah ah, you’re right, my bad. I didn’t look properly at how the
Semaphoremethods differ.SmallRye Fault Tolerance has to work in the actual real world, which will take years to fully adopt virtual threads. Sorry, but this is just the sad state of reality.
Actually one more thing, I probably didn’t express that clearly above. The usage of
CompletionStagein SmallRye Fault Tolerance doesn’t mean you’re required to use non-blocking concurrency. If you do thread offloads (either manually, or ask SmallRye Fault Tolerance to do it for you), you can use blocking concurrency just fine. In that case, you will only create already completeCompletionStages (CompletableFuture.completedStage()andfailedStage()). If you want to block on theCompletionStagereturned by SmallRye Fault Tolerance, you can calltoCompletableFuture(), and that’s guaranteed to work. This indeed entails certain amount of code overhead, but not very big I believe.Ah I see. If you have 20 threads competing for a semaphore that only hands out 2 permits, you indeed have an implicit form of queueing, but at the cost of having 20 threads. That may be fine with virtual threads, but it definitely is not fine with platform threads.
Now, when it comes to
that’s not true. You don’t need the timeout there, that doesn’t really change anything when it comes to the queueing semantics. It is an extra cherry on top.
In SmallRye Fault Tolerance, bulkheads implement an explicit form of queueing. Tasks are put into an actual queue and are not submitted to an executor until they obtain a permit. (Aside: in the annotation-based API, this is only true for
CompletionStage-returning tasks and not forFuture-returning tasks, but the programmatic API doesn’t supportFuture-returning tasks at all.)Ah OK, so you want the queued tasks to have a timeout? And if the task is in the queue for too long, it should fail immediately? That seems like a reasonable request.
Don’t let anyone tell you that virtual threads are some form of a panacea; they are still just threads. The only thing that might possibly change in the future is that we will no longer need to deal with non-blocking concurrency (aka event loops or “reactive”); blocking concurrency (aka multi-threading) will be enough. But we’re certainly not in that world yet; it will take years for the ecosystem to adapt.
Some people equate the word “asynchronous” with one or the other form of concurrency I mentioned above. In SmallRye Fault Tolerance, I use the word “asynchronous” in a more abstract manner, close to the dictionary definition of “not happening at the same time”.
If an operation is asynchronous, it means its result is not available right here right now; the idea of reifying that concept into a future is decades old. As it turns out, if we restrict ourselves to the JDK, the best type for representing futures is currently
CompletionStage/CompletableFuture, because it supports both styles of concurrency. So that’s what SmallRye Fault Tolerance uses. Maybe at some point, this won’t be necessary, but as I mentioned above, we’re years away from that.Now, here’s a piece of code that uses bulkhead with a queue (which requires the operation to be asynchronous, otherwise there’s no way to implement queueing):
Apparently, returning
nullworks in this particular case, but I wouldn’t rely on it for anything real. I really would returnCompletableFuture.completedStage(null);.If I rely on SmallRye Fault Tolerance to submit the tasks to an executor on its own, instead of doing it manually, that code becomes:
This works perfectly well.
You want something like
Then, your task needs to return
CompletionStage, which shouldn’t be an issue, you’d just replacereturn null;withreturn CompletableFuture.completedStage(null);.If all tasks in the application are submitted to the same executor, you could even add
withThreadOffload(true)to theFaultTolerancebuilder and set that executor usingStandaloneFaultTolerance.configure(). That way, you wouldn’t even have to create theRunnableand submit it to the executor manually, SmallRye Fault Tolerance would do that for you. (Actually, this is quite a limitation. At least in the programmatic API, we should be able to specify the executor for eachFaultToleranceobject separately. Let me file an issue for that.)