camel-quarkus: [Camel 4.2] perf-regression: mean throughput regression appears with Camel Quarkus 3.5.0

Bug description

A mean throughput performance drop has been captured on 10 minutes long runs , and it’s confirmed on another machine with 1 hour long runs:

                             Camel Quarkus Throughput Performance Increase Compared to Previous Version                              
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]    |  Native req/s [%increase]  |               Status               |
-------------------------------------------------------------------------------------------------------------------------------------
                 3.4.0  |       10m  |   40631.67 req/s [+0.00%]  |   22104.40 req/s [+0.00%]  |                                OK  |
                 3.5.0  |       10m  |  27947.75 req/s [-31.22%]  |  14816.30 req/s [-32.97%]  |  Potential performance regression  |

                             Camel Quarkus Throughput Performance Increase Compared to Previous Version                              
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]    |  Native req/s [%increase]  |               Status               |
-------------------------------------------------------------------------------------------------------------------------------------
                 3.4.0  |        1h  |   40683.59 req/s [+0.00%]  |   22635.90 req/s [+0.00%]  |                                OK  |
                 3.5.0  |        1h  |  28109.76 req/s [-30.91%]  |  15045.40 req/s [-33.53%]  |  Potential performance regression  |

About this issue

Original URL
State: closed
Created 8 months ago
Comments: 25 (25 by maintainers)

Most upvoted comments

@orpiske I nailed it as explained in #5548. It’s just a flaw in the perf-regression tool. Long story short, the native images were not built with the same jdk version. After forcing the native builder image version, there is no more regression detected 🥳

Many thanks for helping in this investigation. I’m convinced of the positive impact. At the end of the day, it’s yet another performance regression that will not be seen by users of LTS versions 👏

aldettinger on Nov 23, 2023

I am curious about one thing, though (maybe related to what @jamesnetherton mentioned): is this regression specific to one JVM version? If yes, we may have it on Core and I may need to investigate that.

@orpiske So no regression in Camel Quarkus, that’s the beauty of having test setup when only a few things change at a time. That way we ignore JDK and Graalvm bugs that are not under our control.

aldettinger on Nov 23, 2023

Maybe I misunderstand, does that mean we have a performance regression when switching the native builder from JDK 17 to JDK 21?

To be precise, the comparison occurred between:

Camel Quarkus 3.4.0 with Quarkus native-image plugin on native-image 17.0.9 2023-10-17
Camel Quarkus 3.6.0-SNAPHOT@commit with Quarkus native-image plugin on MANDREL 23.1.1.0 JDK 21.0.1+12-LTS

It’s like comparing eggs and chicken ( bad example maybe 😉. So we should not assume any continuity in the native builder image chosen by Quarkus, it can be totally different across versions.

aldettinger on Nov 23, 2023