FusionCache: [BUG] Redis timeouts when upgrading from 2.0.0 to 2.0.1/2.1.0
Creating separate issue from discussion: https://github.com/ZiggyCreatures/FusionCache/issues/423#issuecomment-2772494525
Describe the bug
When upgrading FusionCache from 2.0.0 to 2.0.1/2.1.0, we observe frequent 5 second timeouts and significantly delayed requests.
I’ve isolated the issue to upgrading FusionCache nugets from 2.0.0 to 2.0.1/2.1.0, and locking down the SE.Redis version via Directory.Packages.props.
I’m not able to reproduce by upgrading SE.Redis alone. I am able to reproduce by upgrading FusionCache alone, forcing SE.Redis to specific versions. I confirm in logs the exact version of SE.Redis used when deployed and that this is used by FusionCache in the operations that time out.
- ✅ FusionCache 1.4.1 and StackExchange.Redis 2.8.14
- ✅ FusionCache 1.4.1 and StackExchange.Redis 2.8.22
- ✅ FusionCache 1.4.1 and StackExchange.Redis 2.8.31
- ✅ FusionCache 2.0.0 and StackExchange.Redis 2.8.22
- ❌ FusionCache 2.0.1 and StackExchange.Redis 2.8.22
- ❌ FusionCache 2.1.0 and StackExchange.Redis 2.8.14 (FusionCache 2.1.0 wants >= 2.8.22 though)
- ❌ FusionCache 2.1.0 and StackExchange.Redis 2.8.31
StackExchange.Redis upgrade changelog from 2.8.14 to 2.8.31 does not include any obvious changes related to timeouts, and by pinning the SE.Redis version while upgrading FusionCache we should have isolated the bug to FusionCache.
To Reproduce
None yet, struggling to create a local repro. Can consistently repro in Dev Kubernetes.
Synthetic factory timeouts seems related. Only 3 cache key patterns are affected and all of them use a common “renew in background” setup with very short, soft factory timeout:
- Redis timeouts are 5000 ms
FactorySoftTimeout= 100ms for these failing cache keys- Default fusion cache setup for most of our cache keys, including the 3 failing keys
IsFailSafeEnabled= trueFailSafeMaxDuration= 24hDuration= 5mJitterMaxDuration= 30sAllowBackgroundDistributedCacheOperations= trueDistributedCacheSoft/HardTimeout= 500ms / 3000msReThrowSerializationExceptions= false
Expected behavior
No timeouts.
Versions
I’ve encountered this issue on:
- FusionCache version
2.1.0(but not on2.0.0or1.4.1) - .NET version
8.0.407and9.0.202 - OS version
Ubuntu 22.04.5 LTS(OS image5.15.0-1082-azurein Azure Kubernetes) - bitnami/redis/7.4.2-debian-12-r6 in Azure Kubernetes, using Redis Sentinel mode with 3 Redis nodes
FusionCache nugets:
ZiggyCreatures.FusionCacheZiggyCreatures.FusionCache.Backplane.StackExchangeRedisZiggyCreatures.FusionCache.OpenTelemetryZiggyCreatures.FusionCache.Serialization.SystemTextJson
Additional context
Example log event
{
"exception": {
"StackTraceString": " at ZiggyCreatures.Caching.Fusion.Internals.Distributed.DistributedCacheAccessor.TryGetEntry[TValue](String operationId, String key, FusionCacheEntryOptions options, Boolean hasFallbackValue, Nullable`1 timeout, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/Internals/Distributed/DistributedCacheAccessor_Sync.cs:line 179\n at ZiggyCreatures.Caching.Fusion.FusionCache.TryUpdateMemoryEntryFromDistributedEntry[TValue](String operationId, String key, FusionCacheMemoryEntry`1 memoryEntry) in /_/src/ZiggyCreatures.FusionCache/FusionCache_Sync.cs:line 1230",
"Message": "An error occurred while working with the distributed cache",
"HResult": -2146233088,
"ClassName": "ZiggyCreatures.Caching.Fusion.FusionCacheDistributedCacheException",
"InnerException": {
"StackTraceString": " at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server, T defaultValue) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2099\n at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 484\n at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData)\n at ZiggyCreatures.Caching.Fusion.Internals.RunUtils.RunSyncFuncWithTimeout[TResult](Func`2 syncFunc, TimeSpan timeout, Boolean cancelIfTimeout, Action`1 timedOutTaskProcessor, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/Internals/RunUtils.cs:line 252\n at ZiggyCreatures.Caching.Fusion.Internals.Distributed.DistributedCacheAccessor.TryGetEntry[TValue](String operationId, String key, FusionCacheEntryOptions options, Boolean hasFallbackValue, Nullable`1 timeout, CancellationToken token)",
"Message": "Timeout performing HMGET (5000ms), next: HMGET v2:User:079705e6-ae67-4f79-a5b0-231466fd5081:Images, inst: 125, qu: 0, qs: 125, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 62457, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 236, async-ops: 1948, serverEndpoint: redis-node-1.redis-headless.redis.svc.cluster.local:6379, conn-sec: 937.13, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: xxx-api-6945fb4d87-cpsfb(SE.Redis-v2.8.22.13109), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=31,Free=32736,Min=1,Max=32767), POOL: (Threads=31,QueuedItems=212,CompletedItems=64946,Timers=48), v: 2.8.22.13109 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)",
"commandStatus": "sent",
"HResult": -2146233083,
"ClassName": "StackExchange.Redis.RedisTimeoutException",
"Data": {
"Redis-Version": "2.8.22.13109",
"request-sent-status": "sent",
"Redis-ThreadPool-Items": "(Threads=31,QueuedItems=212,CompletedItems=64946,Timers=48)",
"Redis-Outbound-Pipe-Bytes": "0",
"Redis-Read-State": "ReadAsync",
"Redis-Server-Endpoint": "redis-node-1.redis-headless.redis.svc.cluster.local:6379",
"Redis-ThreadPool-Workers": "(Busy=31,Free=32736,Min=1,Max=32767)",
"Redis-ThreadPool-IO-Completion": "(Busy=0,Free=1000,Min=1,Max=1000)",
"Redis-Message": "HMGET v2:User:88c14bab-10f3-4dbf-b797-5e471d3322fc:ContractorProfileStatistics",
"Redis-Manager": "10 of 10 available",
"Redis-Busy-Workers": "31",
"Redis-Queue-Awaiting-Response": "125",
"Redis-Write-State": "Idle",
"Redis-Server-Connected-Seconds": "937.13",
"Redis-Message-Next": "HMGET v2:User:079705e6-ae67-4f79-a5b0-231466fd5081:Images",
"Redis-Multiplexer-Connects": "1/1/0",
"Redis-OpsSinceLastHeartbeat": "125",
"Redis-Last-Result-Bytes": "0",
"redis-command": "HMGET v2:User:88c14bab-10f3-4dbf-b797-5e471d3322fc:ContractorProfileStatistics",
"Redis-Backlog-Writer": "SpinningDown",
"Redis-Inbound-Bytes": "62457",
"redis-server": "redis-node-1.redis-headless.redis.svc.cluster.local:6379",
"Redis-Inbound-Buffer-Bytes": "0",
"Redis-Client-Name": "xxx-api-6945fb4d87-cpsfb(SE.Redis-v2.8.22.13109)",
"Redis-Inbound-Pipe-Bytes": "0",
"Redis-Active-Writer": "false",
"Redis-Abort-On-Connect": "0",
"Redis-Sync-Ops": "236",
"Redis-Queue-Awaiting-Write": "0",
"Redis-Async-Ops": "1948"
},
"RemoteStackIndex": 0,
"HelpURL": "https://stackexchange.github.io/StackExchange.Redis/Timeouts",
"Source": "StackExchange.Redis"
},
"RemoteStackIndex": 0,
"Source": "ZiggyCreatures.FusionCache"
},
"logzio-signature": -1554305481,
"level": "Error",
"logger": "ZiggyCreatures.Caching.Fusion.FusionCache",
"CacheOperationId": "0HNBI30KS3OVM",
"_logzio_logceptions": [
"a639d9e1e3c8f6d644627d38e75d19fc"
],
"type": "app",
"message": "FUSION [N=\"FusionCache\" I=\"0HNBI30KS3LTI\"] (O=\"0HNBI30KS3OVM\" K=\"User:88c14bab-10f3-4dbf-b797-5e471d3322fc:ContractorProfileStatistics\"): an error occurred while trying to update a memory entry from a distributed entry",
"K8S_NODE_NAME": "aks-ds3v2-38460103-vmss000001",
"CacheInstanceId": "0HNBI30KS3LTI",
"tags": [
"_logz_http_bulk_json_8070"
],
"environment": "",
"RoleName": "XXX.Api",
"LogSize": 5468,
"@timestamp": "2025-04-02T19:12:45.1224358+00:00",
"_logzio_pattern": 2954900,
"Version": "2025.4.2.1",
"K8S_POD_NAMESPACE": "ns-dev2",
"service": "",
"RoleInstance": "xxx-api-6945fb4d87-cpsfb",
"Environment": "K8SDev",
"CacheName": "FusionCache",
"CacheKey": "User:88c14bab-10f3-4dbf-b797-5e471d3322fc:ContractorProfileStatistics",
"messageTemplate": "FUSION [N={CacheName} I={CacheInstanceId}] (O={CacheOperationId} K={CacheKey}): an error occurred while trying to update a memory entry from a distributed entry"
}
}
Stacktrace:
ZiggyCreatures.Caching.Fusion.FusionCacheDistributedCacheException: An error occurred while working with the distributed cache
at ZiggyCreatures.Caching.Fusion.Internals.Distributed.DistributedCacheAccessor.TryGetEntry[TValue](String operationId, String key, FusionCacheEntryOptions options, Boolean hasFallbackValue, Nullable`1 timeout, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/Internals/Distributed/DistributedCacheAccessor_Sync.cs:line 179
at ZiggyCreatures.Caching.Fusion.FusionCache.TryUpdateMemoryEntryFromDistributedEntry[TValue](String operationId, String key, FusionCacheMemoryEntry`1 memoryEntry) in /_/src/ZiggyCreatures.FusionCache/FusionCache_Sync.cs:line 1230
StackExchange.Redis.RedisTimeoutException: Timeout performing HMGET (5000ms), next: HMGET v2:User:079705e6-ae67-4f79-a5b0-231466fd5081:Images, inst: 125, qu: 0, qs: 125, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 62457, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 236, async-ops: 1948, serverEndpoint: redis-node-1.redis-headless.redis.svc.cluster.local:6379, conn-sec: 937.13, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: vaskehjelp-app-api-6945fb4d87-cpsfb(SE.Redis-v2.8.22.13109), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=31,Free=32736,Min=1,Max=32767), POOL: (Threads=31,QueuedItems=212,CompletedItems=64946,Timers=48), v: 2.8.22.13109 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server, T defaultValue) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2099
at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 484
at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData)
at ZiggyCreatures.Caching.Fusion.Internals.RunUtils.RunSyncFuncWithTimeout[TResult](Func`2 syncFunc, TimeSpan timeout, Boolean cancelIfTimeout, Action`1 timedOutTaskProcessor, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/Internals/RunUtils.cs:line 252
at ZiggyCreatures.Caching.Fusion.Internals.Distributed.DistributedCacheAccessor.TryGetEntry[TValue](String operationId, String key, FusionCacheEntryOptions options, Boolean hasFallbackValue, Nullable`1 timeout, CancellationToken token)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 46
I have not compared performance head-to-head with 1.4.1/2.0.0 vs 2.2.0-preview1, but yes, at least timeouts and the major perf impact seems fixed in 2.2.0-preview1 👌
I have a small minimal repro project more or less ready, will share tomorrow.
I have not yet been able to reproduce outside of an API request context, for example in an unit test with multiple FusionCache instances with separate Redis connections and multiplexers and concurrently manipulating cache in the same way the APIs do.
It’s weird, almost like the request context or aspnetcore task scheduling affects this whole thing somehow.
I’m able to repro locally with multiple instances and highly concurrent load. Still need to narrow it down to a simpler repro.
On our side, calling GetOrSet either sequentially or concurrently for a list of user IDs, on images and stats cache keys.
Interesting:
So, it seems multiple API nodes is required to reproduce in Dev. This explains why I’m not seeing this locally, so I’ll try multiple nodes locally and see if I can reproduce.
With verbose FusionCache logs, there is indeed a LOT of activity going on with backplane updates. We’re updating several hundred cache entries in a short time.
Update: Test nr 1 was a bit inconclusive. I confused myself a bit, some API endpoints take a single user ID and some process in bulk. The bulk one did not have any nesting, but would get a lot less cache activity. I’ll test more locally.
Still on 2.1, I’m testing things out in steps.
Nice, added it to my list of things to try.
3 api instances, and we have multiple apis but timeouts are only occurring on a select few endpoints on this one API. The common denominator for the cache access that result in timeouts are:
I just realized that this may just be an issue with our environment. Our Azure Functions runs in an asev3 environment which is inside a vnet, which is likely why it cannot connect. I am not sure what the environment of the original poster is, but in my case I don’t think it’s FusionCache’s fault. Sorry about the false alarm.
Thanks!
Read here: the plan is that 2.1 has a corresponding 2.0.1 that is LTS only, 2.2 will have a corresponding 2.0.2 and so on. This will go on until November, when .NET 10 will come out and we will all be able to move on from this (necessary) mess.
Thoughts?
I am also unable to reproduce locally, but it is a problem once I deploy the code to our Azure Functions QA environment.
Still not able to reproduce locally:
I can’t get any timeouts at all in the logs, and all API responses behave as expected and have no unexpected delays in the tracing.
Things I’ve tried to reproduce:
Task.Delayin strategic places of factories, as well as no delayThings to try in Dev:
Things to try to reproduce locally:
So no news so far really, just more progress on the repro attempts and still more ideas to try out.
I asked a robot for help given the JSON exception and the annotated image, and I think it gave a good summary.
Basically, the thread pool is not able to process all the responses quickly enough and results in timeouts even if maybe Redis itself is not very busy, since the same usage pattern works well with FusionCache 2.0.0.
The robot is a bit biased to my text in the annotated image, because I am no longer convinced nested FusionCache getorset is the root cause. I think this may rather be a symptom of blocked/slow Redis operations.
Is it possible that FusionCache 2.1.0 introduces more synchronous/atomic Redis operations than before?
It seems to me high load and/or many concurrent operations is relevant, in our case refreshing cache for several hundred user profiles at the same time. I also think soft factory timeout is relevant in terms of allowing more concurrency, but we definitely have soft timeouts for cache keys that don’t experience timeouts too.
Key properties of a timeout exception indicated many workers and many queued items:
Analysis of Redis Timeouts and Performance Degradation
Problem:
StackExchange.Redis.RedisTimeoutExceptionerrors are occurring, notably duringHMGEToperations.qsvalues (commands sent to Redis but awaiting response) and critically high.NET ThreadPoolQueuedItems(e.g.,QueuedItems=212), pointing strongly towards client-side ThreadPool starvation.Investigation & Findings (Jaeger Trace):
GetOrSet profile) revealed a significant performance issue within the associatedProfile factorymethod.set to cache leveloperation, presumably writing fetched image data to the Redis cache.Profile factory, which is itself invoked by the parentGetOrSet profilecache call. This confirms a nested caching pattern.Diagnosis:
The root cause appears to be .NET ThreadPool starvation within the client application. This is likely triggered or significantly worsened by:
GetOrSet/Setoperations within a factory method that is already part of a parentGetOrSetoperation.set to cache leveloperation (whether slow due to Redis, large data, or client-side blocking) ties up worker threads..Result,.Wait()) within the async factory method.This ties up available worker threads, preventing the application from promptly processing responses from Redis (causing timeouts) or handling other concurrent work, leading to the high
QueuedItemscount.Recommendations:
Profile factory(High Priority):async/awaitis used correctly throughout the factory. Remove all instances of.Result,.Wait(), etc.GetOrSet profilecall handle the sole responsibility of caching the object returned by the factory.SLOWLOG,COMMANDSTATSfor SET/HMSET latency, CPU/Memory usage onredis-node-1...).QueuedItems, busy threads) and Redis command latency after implementing changes to verify improvement.@jodydonetti Absolutely, we can work around this, but I’m still trying to reproduce locally to hopefully understand the problem better and hopefully get a minimal repro for you.