OrchardCore: Slow requests for static Media files when a lot of them are requested at the same time under Azure App Services (Web Apps)

Describe the bug

We’re seeing strangely slow requests for static Media files. This can’t be explained by slow storage, and while both sites I’ve seen these on use Azure Blob Storage, I ruled out that being slow (and the requests are also slow if the files are already cached on the server’s file system).

Perhaps related: https://github.com/OrchardCMS/OrchardCore/issues/1634.

To Reproduce

I don’t have a 100% repro, but this seems to be the rough playbook when the issue happens:

1. Visit a page that has a lot of images from the Media Library, e.g. https://ikwileentaart.nl/broodjes or https://ikwileentaart.nl/brood.
2. Sometimes observe that images load in 1-2 or frequently even 10s, despite their <100 KB size not warranting this. Not just the client-side load time is slow, but the actual server request being served too, as indicated by Application Insights.

Note that while the linked pages use image resizing, I’ve seen exactly the same issue on another site that loads images without any resizing.

The issue seems to correlate with increased CPU usage, but well below the server’s capacity (<10%). Other metrics are either uncorrelated/normal or show the effect (like increased client receive time), not a probable cause. So, it doesn’t seem that simply there are too many requests coming in and the server is too low spec to handle it.

I ruled out recent shell restarts with tracing. I added tracing to measure the runtime of the bodies of IMediaFileStore.GetFileInfoAsync(string path), GetFileStreamAsync(string path), GetFileStreamAsync(IFileStoreEntry fileStoreEntry), and IMediaFileStoreCacheFileProvider.IsCachedAsync(string path), SetCacheAsync(Stream stream, IFileStoreEntry fileStoreEntry, CancellationToken cancellationToken). Nothing was slow enough (at most ~200ms sometimes, but nothing in the order of magnitude of seconds, let alone 10s of seconds).

Something seems to throttle requests. Perhaps the lazy workers?

Expected behavior

Static Media files are served within milliseconds of the underlying storage’s latency, so for locally cached files <1ms.