github-workflows-kt: [Bug] Requesting maven-metadata.xml becomes slower and slower with each request
Component
- github-workflows-kt (library with the DSL)
- bindings server (https://bindings.krzeminski.it)
- I don’t know
Problem
While trying to solve this exact problem by implementing #1899, I have seen that this is not actually the problem it seems. I started a local server without the prefetch and did run the workflow script mentioned in that other ticket against it. I did 10 runs, with in-between evicting Maven Local and clearing the cache (with an added route to invalidate the cache). Without the pre-fetch:
real 2m1.172s
real 1m16.028s
real 1m20.947s
real 2m36.726s
real 5m49.400s
real 8m7.732s
real 10m27.326s
real 13m5.050s
real 15m29.149s
real 1m58.256s
With the pre-fetch:
real 1m2.741s
real 0m33.011s
real 0m40.233s
real 1m4.125s
real 3m10.570s
real 5m49.260s
real 8m12.268s
real 10m25.164s
real 12m21.703s
real 15m23.932s
There seems to be some problem slowing down things. I first thought it might be GitHub slowing down requests, but as you can see, after server restart (to add the pre-fetch) it was fast again and then again becoming slower and slower. This makes me think the slow-down is cause by something else within the binding server.
What becomes slower and slower is the maven-metadata.xml building.
I filtered the log output of the pre-fetch run for requests that needed more than 1 second. The output is
2025-04-16 14:47:41,076 <INFO > <eventLoopGroupProxy-4-12 > <[]> <{request-id=9a8m8y3rxf7i0mx}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__save___major/maven-metadata.xml in 21024ms
2025-04-16 14:48:00,762 <INFO > <eventLoopGroupProxy-4-12 > <[]> <{request-id=ukcwinp48bw8vau}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__save___major/maven-metadata.xml.sha1 in 19682ms
2025-04-16 14:48:22,117 <INFO > <eventLoopGroupProxy-4-12 > <[]> <{request-id=ekneyzq8zhiei63}> < io.ktor.server.Application> 200 OK: GET - /actions/upload-artifact___major/maven-metadata.xml in 19518ms
2025-04-16 14:48:43,481 <INFO > <eventLoopGroupProxy-4-12 > <[]> <{request-id=1tw2mcbqqy23sju}> < io.ktor.server.Application> 200 OK: GET - /actions/upload-artifact___major/maven-metadata.xml.sha1 in 21347ms
2025-04-16 14:49:06,193 <INFO > <eventLoopGroupProxy-4-12 > <[]> <{request-id=uhejruqgtsqpf5d}> < io.ktor.server.Application> 200 OK: GET - /mxschmitt/action-tmate___major/maven-metadata.xml in 19853ms
2025-04-16 14:49:25,975 <INFO > <eventLoopGroupProxy-4-13 > <[]> <{request-id=0xhk2ddzxcfnrfb}> < io.ktor.server.Application> 200 OK: GET - /mxschmitt/action-tmate___major/maven-metadata.xml.sha1 in 19775ms
2025-04-16 14:49:55,986 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=pdff139kzwn4mqa}> < io.ktor.server.Application> 200 OK: GET - /tecolicom/actions-use-homebrew-tools___major/maven-metadata.xml in 19441ms
2025-04-16 14:50:15,105 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=28gzet7f1cl4kl9}> < io.ktor.server.Application> 200 OK: GET - /tecolicom/actions-use-homebrew-tools___major/maven-metadata.xml.sha1 in 19111ms
2025-04-16 14:50:43,517 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=ffh6uxr19y20ymz}> < io.ktor.server.Application> 200 OK: GET - /Wandalen/wretry.action__main___major/maven-metadata.xml in 27349ms
2025-04-16 14:51:09,365 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=w9f8mzacl3dom05}> < io.ktor.server.Application> 200 OK: GET - /Wandalen/wretry.action__main___major/maven-metadata.xml.sha1 in 25843ms
2025-04-16 14:51:39,392 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=bk2rkvcf1cntuk7}> < io.ktor.server.Application> 200 OK: GET - /actions/checkout___major/maven-metadata.xml in 25933ms
2025-04-16 14:52:04,074 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=8p2zy2o7vpg1nqy}> < io.ktor.server.Application> 200 OK: GET - /actions/checkout___major/maven-metadata.xml.sha1 in 24679ms
2025-04-16 14:52:31,589 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=aboxgsvz83vbcsc}> < io.ktor.server.Application> 200 OK: GET - /actions/setup-java___major/maven-metadata.xml in 25202ms
2025-04-16 14:52:55,150 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=65l2czvfe9d5rbi}> < io.ktor.server.Application> 200 OK: GET - /actions/setup-java___major/maven-metadata.xml.sha1 in 23556ms
2025-04-16 14:53:25,280 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=gqlpg6dsws7vg3l}> < io.ktor.server.Application> 200 OK: GET - /actions/cache___major/maven-metadata.xml in 27737ms
2025-04-16 14:53:50,563 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=cybpjfhhvvqwz9c}> < io.ktor.server.Application> 200 OK: GET - /actions/cache___major/maven-metadata.xml.sha1 in 25279ms
2025-04-16 14:54:20,392 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=zd2kwe5we5zuv0y}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__restore___major/maven-metadata.xml in 27962ms
2025-04-16 14:54:47,601 <INFO > <eventLoopGroupProxy-4-15 > <[]> <{request-id=eqt33mc1x42bf75}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__restore___major/maven-metadata.xml.sha1 in 27204ms
2025-04-16 14:55:18,972 <INFO > <eventLoopGroupProxy-4-16 > <[]> <{request-id=iis9mfqe5roikht}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__save___major/maven-metadata.xml in 29562ms
2025-04-16 14:55:47,072 <INFO > <eventLoopGroupProxy-4-16 > <[]> <{request-id=s0p99im2qlk3l46}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__save___major/maven-metadata.xml.sha1 in 28097ms
2025-04-16 14:56:15,081 <INFO > <eventLoopGroupProxy-4-16 > <[]> <{request-id=kwze0wspfn7056s}> < io.ktor.server.Application> 200 OK: GET - /actions/upload-artifact___major/maven-metadata.xml in 26184ms
2025-04-16 14:56:43,181 <INFO > <eventLoopGroupProxy-4-16 > <[]> <{request-id=dl393jchj7g4ps3}> < io.ktor.server.Application> 200 OK: GET - /actions/upload-artifact___major/maven-metadata.xml.sha1 in 28097ms
2025-04-16 14:57:12,596 <INFO > <eventLoopGroupProxy-4-16 > <[]> <{request-id=8d7gldthlpmfz8j}> < io.ktor.server.Application> 200 OK: GET - /mxschmitt/action-tmate___major/maven-metadata.xml in 25393ms
2025-04-16 14:57:38,807 <INFO > <eventLoopGroupProxy-4-16 > <[]> <{request-id=pi26mdg3fu1ks1f}> < io.ktor.server.Application> 200 OK: GET - /mxschmitt/action-tmate___major/maven-metadata.xml.sha1 in 26208ms
2025-04-16 14:58:15,213 <INFO > <eventLoopGroupProxy-4-18 > <[]> <{request-id=4b645ljcbgacvsp}> < io.ktor.server.Application> 200 OK: GET - /tecolicom/actions-use-homebrew-tools___major/maven-metadata.xml in 26327ms
2025-04-16 14:58:40,269 <INFO > <eventLoopGroupProxy-4-18 > <[]> <{request-id=7zl76l182w6phac}> < io.ktor.server.Application> 200 OK: GET - /tecolicom/actions-use-homebrew-tools___major/maven-metadata.xml.sha1 in 25050ms
2025-04-16 14:59:13,169 <INFO > <eventLoopGroupProxy-4-18 > <[]> <{request-id=mf585b5r65l70jo}> < io.ktor.server.Application> 200 OK: GET - /Wandalen/wretry.action__main___major/maven-metadata.xml in 31843ms
2025-04-16 14:59:43,654 <INFO > <eventLoopGroupProxy-4-18 > <[]> <{request-id=jw69w8ob0ymh8ul}> < io.ktor.server.Application> 200 OK: GET - /Wandalen/wretry.action__main___major/maven-metadata.xml.sha1 in 30481ms
2025-04-16 15:00:16,461 <INFO > <eventLoopGroupProxy-4-18 > <[]> <{request-id=wwur7j741qk6fej}> < io.ktor.server.Application> 200 OK: GET - /actions/checkout___major/maven-metadata.xml in 28721ms
2025-04-16 15:00:46,961 <INFO > <eventLoopGroupProxy-4-18 > <[]> <{request-id=w5xm5dlamhlqtqz}> < io.ktor.server.Application> 200 OK: GET - /actions/checkout___major/maven-metadata.xml.sha1 in 30497ms
2025-04-16 15:01:19,712 <INFO > <eventLoopGroupProxy-4-18 > <[]> <{request-id=azm3pckdi02ddn3}> < io.ktor.server.Application> 200 OK: GET - /actions/setup-java___major/maven-metadata.xml in 30412ms
2025-04-16 15:01:51,367 <INFO > <eventLoopGroupProxy-4-18 > <[]> <{request-id=6276bx7gmnswlum}> < io.ktor.server.Application> 200 OK: GET - /actions/setup-java___major/maven-metadata.xml.sha1 in 31651ms
2025-04-16 15:02:29,462 <INFO > <eventLoopGroupProxy-4-18 > <[]> <{request-id=h1jwmf2yrl0pibc}> < io.ktor.server.Application> 200 OK: GET - /actions/cache___major/maven-metadata.xml in 35567ms
2025-04-16 15:03:02,440 <INFO > <eventLoopGroupProxy-4-18 > <[]> <{request-id=3r8gwai6mg2aasd}> < io.ktor.server.Application> 200 OK: GET - /actions/cache___major/maven-metadata.xml.sha1 in 32971ms
2025-04-16 15:03:38,909 <INFO > <eventLoopGroupProxy-4-19 > <[]> <{request-id=bwkizkvn7knlo4p}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__restore___major/maven-metadata.xml in 34668ms
2025-04-16 15:04:19,162 <INFO > <eventLoopGroupProxy-4-19 > <[]> <{request-id=z7f4u2xc8hcz5j7}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__restore___major/maven-metadata.xml.sha1 in 40238ms
2025-04-16 15:05:02,218 <INFO > <eventLoopGroupProxy-4-19 > <[]> <{request-id=poumvwcha9hyx6u}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__save___major/maven-metadata.xml in 41200ms
2025-04-16 15:05:42,941 <INFO > <eventLoopGroupProxy-4-19 > <[]> <{request-id=3jtpyu49tvyfvpa}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__save___major/maven-metadata.xml.sha1 in 40718ms
2025-04-16 15:06:22,856 <INFO > <eventLoopGroupProxy-4-19 > <[]> <{request-id=4ghn8t1arnb7bz6}> < io.ktor.server.Application> 200 OK: GET - /actions/upload-artifact___major/maven-metadata.xml in 38056ms
2025-04-16 15:06:56,126 <INFO > <eventLoopGroupProxy-4-19 > <[]> <{request-id=66b07hu2t8q3i3h}> < io.ktor.server.Application> 200 OK: GET - /actions/upload-artifact___major/maven-metadata.xml.sha1 in 33264ms
2025-04-16 15:07:33,396 <INFO > <eventLoopGroupProxy-4-19 > <[]> <{request-id=mssb56secu152gn}> < io.ktor.server.Application> 200 OK: GET - /mxschmitt/action-tmate___major/maven-metadata.xml in 34319ms
2025-04-16 15:08:03,533 <INFO > <eventLoopGroupProxy-4-19 > <[]> <{request-id=d9z013q8jxfyqx0}> < io.ktor.server.Application> 200 OK: GET - /mxschmitt/action-tmate___major/maven-metadata.xml.sha1 in 30119ms
2025-04-16 15:08:44,167 <INFO > <eventLoopGroupProxy-4-22 > <[]> <{request-id=weast6igcynwjwz}> < io.ktor.server.Application> 200 OK: GET - /tecolicom/actions-use-homebrew-tools___major/maven-metadata.xml in 29823ms
2025-04-16 15:09:13,202 <INFO > <eventLoopGroupProxy-4-22 > <[]> <{request-id=1jnrs3xdkyjvahb}> < io.ktor.server.Application> 200 OK: GET - /tecolicom/actions-use-homebrew-tools___major/maven-metadata.xml.sha1 in 29031ms
2025-04-16 15:09:56,643 <INFO > <eventLoopGroupProxy-4-22 > <[]> <{request-id=dobeybq4qa7fqeg}> < io.ktor.server.Application> 200 OK: GET - /Wandalen/wretry.action__main___major/maven-metadata.xml in 42407ms
2025-04-16 15:10:36,501 <INFO > <eventLoopGroupProxy-4-22 > <[]> <{request-id=sc3l4wyrh838jex}> < io.ktor.server.Application> 200 OK: GET - /Wandalen/wretry.action__main___major/maven-metadata.xml.sha1 in 39853ms
2025-04-16 15:11:19,000 <INFO > <eventLoopGroupProxy-4-22 > <[]> <{request-id=59zkciw1hpbpf6l}> < io.ktor.server.Application> 200 OK: GET - /actions/checkout___major/maven-metadata.xml in 38495ms
2025-04-16 15:11:53,711 <INFO > <eventLoopGroupProxy-4-22 > <[]> <{request-id=oz0itreqmlh71sa}> < io.ktor.server.Application> 200 OK: GET - /actions/checkout___major/maven-metadata.xml.sha1 in 34708ms
2025-04-16 15:12:31,737 <INFO > <eventLoopGroupProxy-4-22 > <[]> <{request-id=co36sk1diax0xzh}> < io.ktor.server.Application> 200 OK: GET - /actions/setup-java___major/maven-metadata.xml in 35466ms
2025-04-16 15:13:08,641 <INFO > <eventLoopGroupProxy-4-22 > <[]> <{request-id=hhy1556mzrky6am}> < io.ktor.server.Application> 200 OK: GET - /actions/setup-java___major/maven-metadata.xml.sha1 in 36890ms
2025-04-16 15:13:51,581 <INFO > <eventLoopGroupProxy-4-22 > <[]> <{request-id=48yi880ppd9kga3}> < io.ktor.server.Application> 200 OK: GET - /actions/cache___major/maven-metadata.xml in 40024ms
2025-04-16 15:14:31,434 <INFO > <eventLoopGroupProxy-4-1 > <[]> <{request-id=8s4bze00kdtik6k}> < io.ktor.server.Application> 200 OK: GET - /actions/cache___major/maven-metadata.xml.sha1 in 39848ms
2025-04-16 15:15:17,951 <INFO > <eventLoopGroupProxy-4-1 > <[]> <{request-id=u67jplno8nx88nn}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__restore___major/maven-metadata.xml in 44660ms
2025-04-16 15:16:01,147 <INFO > <eventLoopGroupProxy-4-1 > <[]> <{request-id=70m8mo721tr51k7}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__restore___major/maven-metadata.xml.sha1 in 43193ms
2025-04-16 15:16:46,571 <INFO > <eventLoopGroupProxy-4-1 > <[]> <{request-id=on7y1vgktl7d1x0}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__save___major/maven-metadata.xml in 43593ms
2025-04-16 15:17:29,027 <INFO > <eventLoopGroupProxy-4-1 > <[]> <{request-id=cnztgrf4dvyvy74}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__save___major/maven-metadata.xml.sha1 in 42452ms
2025-04-16 15:18:14,171 <INFO > <eventLoopGroupProxy-4-1 > <[]> <{request-id=go7ydqnqy6c8wcl}> < io.ktor.server.Application> 200 OK: GET - /actions/upload-artifact___major/maven-metadata.xml in 43311ms
2025-04-16 15:18:58,592 <INFO > <eventLoopGroupProxy-4-1 > <[]> <{request-id=2xp4tmcqffrpzgg}> < io.ktor.server.Application> 200 OK: GET - /actions/upload-artifact___major/maven-metadata.xml.sha1 in 44417ms
2025-04-16 15:19:44,371 <INFO > <eventLoopGroupProxy-4-2 > <[]> <{request-id=hmoriuf6rqlclwi}> < io.ktor.server.Application> 200 OK: GET - /mxschmitt/action-tmate___major/maven-metadata.xml in 42889ms
2025-04-16 15:20:25,706 <INFO > <eventLoopGroupProxy-4-2 > <[]> <{request-id=fmx5gxs71e2lbse}> < io.ktor.server.Application> 200 OK: GET - /mxschmitt/action-tmate___major/maven-metadata.xml.sha1 in 41331ms
2025-04-16 15:21:14,668 <INFO > <eventLoopGroupProxy-4-4 > <[]> <{request-id=ek8qjyp4xwgaugm}> < io.ktor.server.Application> 200 OK: GET - /tecolicom/actions-use-homebrew-tools___major/maven-metadata.xml in 38678ms
2025-04-16 15:21:54,231 <INFO > <eventLoopGroupProxy-4-4 > <[]> <{request-id=ph9k2ae7r2ogm98}> < io.ktor.server.Application> 200 OK: GET - /tecolicom/actions-use-homebrew-tools___major/maven-metadata.xml.sha1 in 39558ms
2025-04-16 15:22:45,352 <INFO > <eventLoopGroupProxy-4-4 > <[]> <{request-id=qsbvyax7nbxyjef}> < io.ktor.server.Application> 200 OK: GET - /Wandalen/wretry.action__main___major/maven-metadata.xml in 50104ms
2025-04-16 15:23:36,806 <INFO > <eventLoopGroupProxy-4-4 > <[]> <{request-id=9k5x4kgjqle6c3m}> < io.ktor.server.Application> 200 OK: GET - /Wandalen/wretry.action__main___major/maven-metadata.xml.sha1 in 51449ms
2025-04-16 15:24:24,365 <INFO > <eventLoopGroupProxy-4-4 > <[]> <{request-id=9bpn0fj1gesbrry}> < io.ktor.server.Application> 200 OK: GET - /actions/checkout___major/maven-metadata.xml in 43516ms
2025-04-16 15:25:09,759 <INFO > <eventLoopGroupProxy-4-4 > <[]> <{request-id=40v27ie0yo78bu9}> < io.ktor.server.Application> 200 OK: GET - /actions/checkout___major/maven-metadata.xml.sha1 in 45390ms
2025-04-16 15:26:02,152 <INFO > <eventLoopGroupProxy-4-4 > <[]> <{request-id=2cszra5yli102sv}> < io.ktor.server.Application> 200 OK: GET - /actions/setup-java___major/maven-metadata.xml in 50105ms
2025-04-16 15:26:49,641 <INFO > <eventLoopGroupProxy-4-5 > <[]> <{request-id=n6fsyb6acl0mdy4}> < io.ktor.server.Application> 200 OK: GET - /actions/setup-java___major/maven-metadata.xml.sha1 in 47482ms
2025-04-16 15:27:49,075 <INFO > <eventLoopGroupProxy-4-5 > <[]> <{request-id=xwf7y94wf51r95k}> < io.ktor.server.Application> 200 OK: GET - /actions/cache___major/maven-metadata.xml in 57035ms
2025-04-16 15:28:42,772 <INFO > <eventLoopGroupProxy-4-5 > <[]> <{request-id=25z67diw6bj7ma7}> < io.ktor.server.Application> 200 OK: GET - /actions/cache___major/maven-metadata.xml.sha1 in 53693ms
2025-04-16 15:29:42,023 <INFO > <eventLoopGroupProxy-4-5 > <[]> <{request-id=t43fqnz4ao5bsbg}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__restore___major/maven-metadata.xml in 57191ms
2025-04-16 15:30:38,042 <INFO > <eventLoopGroupProxy-4-5 > <[]> <{request-id=obnqqds0unr19m7}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__restore___major/maven-metadata.xml.sha1 in 56013ms
2025-04-16 15:31:33,260 <INFO > <eventLoopGroupProxy-4-5 > <[]> <{request-id=dxcs4lzlm112bcv}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__save___major/maven-metadata.xml in 53424ms
2025-04-16 15:32:25,954 <INFO > <eventLoopGroupProxy-4-6 > <[]> <{request-id=8yf8zgtuoqldms5}> < io.ktor.server.Application> 200 OK: GET - /actions/cache__save___major/maven-metadata.xml.sha1 in 52688ms
2025-04-16 15:33:17,764 <INFO > <eventLoopGroupProxy-4-6 > <[]> <{request-id=x7vwpjaqyl57opn}> < io.ktor.server.Application> 200 OK: GET - /actions/upload-artifact___major/maven-metadata.xml in 50013ms
2025-04-16 15:34:07,748 <INFO > <eventLoopGroupProxy-4-6 > <[]> <{request-id=iph66bxcm6zusxl}> < io.ktor.server.Application> 200 OK: GET - /actions/upload-artifact___major/maven-metadata.xml.sha1 in 49980ms
2025-04-16 15:35:01,093 <INFO > <eventLoopGroupProxy-4-6 > <[]> <{request-id=fufwxblrmlpjq78}> < io.ktor.server.Application> 200 OK: GET - /mxschmitt/action-tmate___major/maven-metadata.xml in 50283ms
2025-04-16 15:35:50,006 <INFO > <eventLoopGroupProxy-4-6 > <[]> <{request-id=p5dhliixr9nfxef}> < io.ktor.server.Application> 200 OK: GET - /mxschmitt/action-tmate___major/maven-metadata.xml.sha1 in 48897ms
This shows that the requests to maven-metadata.xml become slower and slower.
Also requesting the checksum for the file becomes slower, as the maven-metadata.xml* generation is currently not cached, so requesting the checksum also regenerates the file from scratch to calculate the checksum. (#1903)
This also happens when running against the official server instance, which made me have the idea for #1899. So it should be investigated what is slowing down theses requests in that extent.
The test I made locally to get the above numbers is
for i in {1..10}; do rm -rf ~/.m2/repository/*/*___major; curl -i -X DELETE http://localhost:8080/cache; echo; time kotlin .github/workflows/build.main.kts; done
with adding
route("cache") {
delete {
bindingsCache.invalidateAll()
call.respondText(text = "OK")
}
}
to ArtifactRoutesKt#artifactRoutes.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 20 (6 by maintainers)
Commits related to this issue
- fix(server): properly handle the lifecycle of ktor-client #1904 — committed to typesafegithub/github-workflows-kt by Vampire 2 years ago
- fix(server): properly handle the lifecycle of ktor-client #1904 — committed to typesafegithub/github-workflows-kt by Vampire 2 years ago
- fix(server): properly handle the lifecycle of ktor-client (#1913) Fixes #1904 When using `ktor-client` you have to manage its lifecycle. That means, if you create a `HttpClientEngine` manually, you... — committed to Vampire/github-workflows-kt by Vampire 2 years ago
I’ll add it.
If Ktor object doesn’t have it by default we’ll have to write code to it
Here a JFR recording of doing another request after that which took 40 seconds and resulted in 500, maybe you can read something from it: https://www.swisstransfer.com/d/87f79762-1695-48a1-a9ed-93a9a0cc5489
Here an even simpler test setup. Against current main (i.e. without cached metadata route), execute
The first 63 requests for me where around 0.7 seconds. Then it started to increase and quickly became over 10 seconds
Added to grafana. Feel free to tweak it around