FusionCache: [BUG] Redis timeouts when upgrading from 2.0.0 to 2.0.1/2.1.0

Creating separate issue from discussion: https://github.com/ZiggyCreatures/FusionCache/issues/423#issuecomment-2772494525

Describe the bug

When upgrading FusionCache from 2.0.0 to 2.0.1/2.1.0, we observe frequent 5 second timeouts and significantly delayed requests.

I’ve isolated the issue to upgrading FusionCache nugets from 2.0.0 to 2.0.1/2.1.0, and locking down the SE.Redis version via Directory.Packages.props.

I’m not able to reproduce by upgrading SE.Redis alone. I am able to reproduce by upgrading FusionCache alone, forcing SE.Redis to specific versions. I confirm in logs the exact version of SE.Redis used when deployed and that this is used by FusionCache in the operations that time out.

  • ✅ FusionCache 1.4.1 and StackExchange.Redis 2.8.14
  • ✅ FusionCache 1.4.1 and StackExchange.Redis 2.8.22
  • ✅ FusionCache 1.4.1 and StackExchange.Redis 2.8.31
  • ✅ FusionCache 2.0.0 and StackExchange.Redis 2.8.22
  • ❌ FusionCache 2.0.1 and StackExchange.Redis 2.8.22
  • ❌ FusionCache 2.1.0 and StackExchange.Redis 2.8.14 (FusionCache 2.1.0 wants >= 2.8.22 though)
  • ❌ FusionCache 2.1.0 and StackExchange.Redis 2.8.31

StackExchange.Redis upgrade changelog from 2.8.14 to 2.8.31 does not include any obvious changes related to timeouts, and by pinning the SE.Redis version while upgrading FusionCache we should have isolated the bug to FusionCache.

To Reproduce

None yet, struggling to create a local repro. Can consistently repro in Dev Kubernetes.

Synthetic factory timeouts seems related. Only 3 cache key patterns are affected and all of them use a common “renew in background” setup with very short, soft factory timeout:

  • Redis timeouts are 5000 ms
  • FactorySoftTimeout = 100ms for these failing cache keys
  • Default fusion cache setup for most of our cache keys, including the 3 failing keys
    • IsFailSafeEnabled = true
    • FailSafeMaxDuration = 24h
    • Duration = 5m
    • JitterMaxDuration = 30s
    • AllowBackgroundDistributedCacheOperations = true
    • DistributedCacheSoft/HardTimeout = 500ms / 3000ms
    • ReThrowSerializationExceptions = false

Expected behavior

No timeouts.

Versions

I’ve encountered this issue on:

  • FusionCache version 2.1.0 (but not on 2.0.0 or 1.4.1)
  • .NET version 8.0.407 and 9.0.202
  • OS version Ubuntu 22.04.5 LTS (OS image 5.15.0-1082-azure in Azure Kubernetes)
  • bitnami/redis/7.4.2-debian-12-r6 in Azure Kubernetes, using Redis Sentinel mode with 3 Redis nodes

FusionCache nugets:

  • ZiggyCreatures.FusionCache
  • ZiggyCreatures.FusionCache.Backplane.StackExchangeRedis
  • ZiggyCreatures.FusionCache.OpenTelemetry
  • ZiggyCreatures.FusionCache.Serialization.SystemTextJson

Additional context

Example log event

{
    "exception": {
      "StackTraceString": "   at ZiggyCreatures.Caching.Fusion.Internals.Distributed.DistributedCacheAccessor.TryGetEntry[TValue](String operationId, String key, FusionCacheEntryOptions options, Boolean hasFallbackValue, Nullable`1 timeout, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/Internals/Distributed/DistributedCacheAccessor_Sync.cs:line 179\n   at ZiggyCreatures.Caching.Fusion.FusionCache.TryUpdateMemoryEntryFromDistributedEntry[TValue](String operationId, String key, FusionCacheMemoryEntry`1 memoryEntry) in /_/src/ZiggyCreatures.FusionCache/FusionCache_Sync.cs:line 1230",
      "Message": "An error occurred while working with the distributed cache",
      "HResult": -2146233088,
      "ClassName": "ZiggyCreatures.Caching.Fusion.FusionCacheDistributedCacheException",
      "InnerException": {
        "StackTraceString": "   at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server, T defaultValue) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2099\n   at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 484\n   at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData)\n   at ZiggyCreatures.Caching.Fusion.Internals.RunUtils.RunSyncFuncWithTimeout[TResult](Func`2 syncFunc, TimeSpan timeout, Boolean cancelIfTimeout, Action`1 timedOutTaskProcessor, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/Internals/RunUtils.cs:line 252\n   at ZiggyCreatures.Caching.Fusion.Internals.Distributed.DistributedCacheAccessor.TryGetEntry[TValue](String operationId, String key, FusionCacheEntryOptions options, Boolean hasFallbackValue, Nullable`1 timeout, CancellationToken token)",
        "Message": "Timeout performing HMGET (5000ms), next: HMGET v2:User:079705e6-ae67-4f79-a5b0-231466fd5081:Images, inst: 125, qu: 0, qs: 125, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 62457, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 236, async-ops: 1948, serverEndpoint: redis-node-1.redis-headless.redis.svc.cluster.local:6379, conn-sec: 937.13, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: xxx-api-6945fb4d87-cpsfb(SE.Redis-v2.8.22.13109), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=31,Free=32736,Min=1,Max=32767), POOL: (Threads=31,QueuedItems=212,CompletedItems=64946,Timers=48), v: 2.8.22.13109 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)",
        "commandStatus": "sent",
        "HResult": -2146233083,
        "ClassName": "StackExchange.Redis.RedisTimeoutException",
        "Data": {
          "Redis-Version": "2.8.22.13109",
          "request-sent-status": "sent",
          "Redis-ThreadPool-Items": "(Threads=31,QueuedItems=212,CompletedItems=64946,Timers=48)",
          "Redis-Outbound-Pipe-Bytes": "0",
          "Redis-Read-State": "ReadAsync",
          "Redis-Server-Endpoint": "redis-node-1.redis-headless.redis.svc.cluster.local:6379",
          "Redis-ThreadPool-Workers": "(Busy=31,Free=32736,Min=1,Max=32767)",
          "Redis-ThreadPool-IO-Completion": "(Busy=0,Free=1000,Min=1,Max=1000)",
          "Redis-Message": "HMGET v2:User:88c14bab-10f3-4dbf-b797-5e471d3322fc:ContractorProfileStatistics",
          "Redis-Manager": "10 of 10 available",
          "Redis-Busy-Workers": "31",
          "Redis-Queue-Awaiting-Response": "125",
          "Redis-Write-State": "Idle",
          "Redis-Server-Connected-Seconds": "937.13",
          "Redis-Message-Next": "HMGET v2:User:079705e6-ae67-4f79-a5b0-231466fd5081:Images",
          "Redis-Multiplexer-Connects": "1/1/0",
          "Redis-OpsSinceLastHeartbeat": "125",
          "Redis-Last-Result-Bytes": "0",
          "redis-command": "HMGET v2:User:88c14bab-10f3-4dbf-b797-5e471d3322fc:ContractorProfileStatistics",
          "Redis-Backlog-Writer": "SpinningDown",
          "Redis-Inbound-Bytes": "62457",
          "redis-server": "redis-node-1.redis-headless.redis.svc.cluster.local:6379",
          "Redis-Inbound-Buffer-Bytes": "0",
          "Redis-Client-Name": "xxx-api-6945fb4d87-cpsfb(SE.Redis-v2.8.22.13109)",
          "Redis-Inbound-Pipe-Bytes": "0",
          "Redis-Active-Writer": "false",
          "Redis-Abort-On-Connect": "0",
          "Redis-Sync-Ops": "236",
          "Redis-Queue-Awaiting-Write": "0",
          "Redis-Async-Ops": "1948"
        },
        "RemoteStackIndex": 0,
        "HelpURL": "https://stackexchange.github.io/StackExchange.Redis/Timeouts",
        "Source": "StackExchange.Redis"
      },
      "RemoteStackIndex": 0,
      "Source": "ZiggyCreatures.FusionCache"
    },
    "logzio-signature": -1554305481,
    "level": "Error",
    "logger": "ZiggyCreatures.Caching.Fusion.FusionCache",
    "CacheOperationId": "0HNBI30KS3OVM",
    "_logzio_logceptions": [
      "a639d9e1e3c8f6d644627d38e75d19fc"
    ],
    "type": "app",
    "message": "FUSION [N=\"FusionCache\" I=\"0HNBI30KS3LTI\"] (O=\"0HNBI30KS3OVM\" K=\"User:88c14bab-10f3-4dbf-b797-5e471d3322fc:ContractorProfileStatistics\"): an error occurred while trying to update a memory entry from a distributed entry",
    "K8S_NODE_NAME": "aks-ds3v2-38460103-vmss000001",
    "CacheInstanceId": "0HNBI30KS3LTI",
    "tags": [
      "_logz_http_bulk_json_8070"
    ],
    "environment": "",
    "RoleName": "XXX.Api",
    "LogSize": 5468,
    "@timestamp": "2025-04-02T19:12:45.1224358+00:00",
    "_logzio_pattern": 2954900,
    "Version": "2025.4.2.1",
    "K8S_POD_NAMESPACE": "ns-dev2",
    "service": "",
    "RoleInstance": "xxx-api-6945fb4d87-cpsfb",
    "Environment": "K8SDev",
    "CacheName": "FusionCache",
    "CacheKey": "User:88c14bab-10f3-4dbf-b797-5e471d3322fc:ContractorProfileStatistics",
    "messageTemplate": "FUSION [N={CacheName} I={CacheInstanceId}] (O={CacheOperationId} K={CacheKey}): an error occurred while trying to update a memory entry from a distributed entry"
  }
}

Stacktrace:

ZiggyCreatures.Caching.Fusion.FusionCacheDistributedCacheException: An error occurred while working with the distributed cache
   at ZiggyCreatures.Caching.Fusion.Internals.Distributed.DistributedCacheAccessor.TryGetEntry[TValue](String operationId, String key, FusionCacheEntryOptions options, Boolean hasFallbackValue, Nullable`1 timeout, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/Internals/Distributed/DistributedCacheAccessor_Sync.cs:line 179
   at ZiggyCreatures.Caching.Fusion.FusionCache.TryUpdateMemoryEntryFromDistributedEntry[TValue](String operationId, String key, FusionCacheMemoryEntry`1 memoryEntry) in /_/src/ZiggyCreatures.FusionCache/FusionCache_Sync.cs:line 1230

StackExchange.Redis.RedisTimeoutException: Timeout performing HMGET (5000ms), next: HMGET v2:User:079705e6-ae67-4f79-a5b0-231466fd5081:Images, inst: 125, qu: 0, qs: 125, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 62457, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 236, async-ops: 1948, serverEndpoint: redis-node-1.redis-headless.redis.svc.cluster.local:6379, conn-sec: 937.13, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: vaskehjelp-app-api-6945fb4d87-cpsfb(SE.Redis-v2.8.22.13109), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=31,Free=32736,Min=1,Max=32767), POOL: (Threads=31,QueuedItems=212,CompletedItems=64946,Timers=48), v: 2.8.22.13109 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
   at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server, T defaultValue) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2099
   at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 484
   at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData)
   at ZiggyCreatures.Caching.Fusion.Internals.RunUtils.RunSyncFuncWithTimeout[TResult](Func`2 syncFunc, TimeSpan timeout, Boolean cancelIfTimeout, Action`1 timedOutTaskProcessor, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/Internals/RunUtils.cs:line 252
   at ZiggyCreatures.Caching.Fusion.Internals.Distributed.DistributedCacheAccessor.TryGetEntry[TValue](String operationId, String key, FusionCacheEntryOptions options, Boolean hasFallbackValue, Nullable`1 timeout, CancellationToken token)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 46

Most upvoted comments

I have not compared performance head-to-head with 1.4.1/2.0.0 vs 2.2.0-preview1, but yes, at least timeouts and the major perf impact seems fixed in 2.2.0-preview1 👌

I have a small minimal repro project more or less ready, will share tomorrow.

I have not yet been able to reproduce outside of an API request context, for example in an unit test with multiple FusionCache instances with separate Redis connections and multiplexers and concurrently manipulating cache in the same way the APIs do.

It’s weird, almost like the request context or aspnetcore task scheduling affects this whole thing somehow.

I’m able to repro locally with multiple instances and highly concurrent load. Still need to narrow it down to a simpler repro.

You mean on your side or i FusionCache?

On our side, calling GetOrSet either sequentially or concurrently for a list of user IDs, on images and stats cache keys.

Interesting:

  1. ✅ No cache for images/stats (no nesting for single user ID endpoints, less cache activity for bulk endpoints) => no timeouts
  2. ❌ Sequential GetOrSet instead of concurrent => fewer timeouts, but still timeouts
  3. ✅ Scale down from 2 to 1 API nodes (nested/concurrent FusionCache) => no timeouts

So, it seems multiple API nodes is required to reproduce in Dev. This explains why I’m not seeing this locally, so I’ll try multiple nodes locally and see if I can reproduce.

With verbose FusionCache logs, there is indeed a LOT of activity going on with backplane updates. We’re updating several hundred cache entries in a short time.

Update: Test nr 1 was a bit inconclusive. I confused myself a bit, some API endpoints take a single user ID and some process in bulk. The bulk one did not have any nesting, but would get a lot less cache activity. I’ll test more locally.

Still on 2.1, I’m testing things out in steps.

Nice, added it to my list of things to try.

3 api instances, and we have multiple apis but timeouts are only occurring on a select few endpoints on this one API. The common denominator for the cache access that result in timeouts are:

  1. Nested FusionCache usages, and
  2. Short soft factory timeouts that intentionally results in many synthetic timeouts.

I just realized that this may just be an issue with our environment. Our Azure Functions runs in an asev3 environment which is inside a vnet, which is likely why it cannot connect. I am not sure what the environment of the original poster is, but in my case I don’t think it’s FusionCache’s fault. Sorry about the false alarm.

I will check it out once it’s ready.

Thanks!

But I must ask, right now I have to use 2.0.1 because I cannot have any dependencies other than .net 8. I already tried 2.0.0 and 2.1.0 and they do not work. How will that work for other versions moving forward?

Read here: the plan is that 2.1 has a corresponding 2.0.1 that is LTS only, 2.2 will have a corresponding 2.0.2 and so on. This will go on until November, when .NET 10 will come out and we will all be able to move on from this (necessary) mess.

Thoughts?

I am also unable to reproduce locally, but it is a problem once I deploy the code to our Azure Functions QA environment.

Still not able to reproduce locally:

  • Running local integration test as repro setup, with real database and real Redis configured
  • Using FusionCache 2.1.0
  • Seeding 500 user profiles, which should be plenty compared to Dev
  • Continuously spams an API endpoint that fetches all user profiles with the nested FusionCache factories as discussed earlier, this endpoint gets frequent timeouts in Dev

I can’t get any timeouts at all in the logs, and all API responses behave as expected and have no unexpected delays in the tracing.

Things I’ve tried to reproduce:

  • Local Redis in Docker
  • Dev Redis in Kubernetes, the same instance we’re seeing timeouts with
  • Task.Delay in strategic places of factories, as well as no delay

Things to try in Dev:

  • Add OpenTelemetry.Instrumentation.StackExchangeRedis, it describes Redis commands and response times
  • ❌ Single multiplexer for cache + backplane (still timeouts)
  • ❌ Remove nesting of FusionCache (still timeouts)
  • Remove soft factory timeouts
  • ✅ Try 2.2.0-preview1 (no more timeouts 🎉 )
  • Increase cpu/memory limits of the API
  • Find more metrics about the health of the Redis instance, maybe it’s overwhelmed

Things to try to reproduce locally:

  • ✅ More concurrency, more load, more API endpoints (can consistently reproduce with 3 API instances)
  • Ubuntu environment in Docker, closer to the Kubernetes environment (currently testing natively on Mac and Windows)

So no news so far really, just more progress on the repro attempts and still more ideas to try out.

I asked a robot for help given the JSON exception and the annotated image, and I think it gave a good summary.

Basically, the thread pool is not able to process all the responses quickly enough and results in timeouts even if maybe Redis itself is not very busy, since the same usage pattern works well with FusionCache 2.0.0.

The robot is a bit biased to my text in the annotated image, because I am no longer convinced nested FusionCache getorset is the root cause. I think this may rather be a symptom of blocked/slow Redis operations.

Is it possible that FusionCache 2.1.0 introduces more synchronous/atomic Redis operations than before?

It seems to me high load and/or many concurrent operations is relevant, in our case refreshing cache for several hundred user profiles at the same time. I also think soft factory timeout is relevant in terms of allowing more concurrency, but we definitely have soft timeouts for cache keys that don’t experience timeouts too.

Key properties of a timeout exception indicated many workers and many queued items:

{
"Redis-ThreadPool-Items": "(Threads=31,QueuedItems=212,CompletedItems=64946,Timers=48)",
"Redis-ThreadPool-Workers": "(Busy=31,Free=32736,Min=1,Max=32767)",
"Redis-Busy-Workers": "31",
"Redis-Queue-Awaiting-Response": "125",  // "qs"
"Redis-Inbound-Bytes": "62457",
"Redis-Sync-Ops": "236",
"Redis-Queue-Awaiting-Write": "0",
"Redis-Async-Ops": "1948"
}

Analysis of Redis Timeouts and Performance Degradation

Problem:

  • Frequent StackExchange.Redis.RedisTimeoutException errors are occurring, notably during HMGET operations.
  • Error logs indicate high qs values (commands sent to Redis but awaiting response) and critically high .NET ThreadPool QueuedItems (e.g., QueuedItems=212), pointing strongly towards client-side ThreadPool starvation.

Investigation & Findings (Jaeger Trace):

  • A Jaeger trace for the profile loading process (GetOrSet profile) revealed a significant performance issue within the associated Profile factory method.
  • The trace shows an extremely long duration (~767ms) for a set to cache level operation, presumably writing fetched image data to the Redis cache.
  • Crucially, this slow cache set operation occurs inside the Profile factory, which is itself invoked by the parent GetOrSet profile cache call. This confirms a nested caching pattern.

Diagnosis:

The root cause appears to be .NET ThreadPool starvation within the client application. This is likely triggered or significantly worsened by:

  1. Nested FusionCache Calls: Performing cache GetOrSet/Set operations within a factory method that is already part of a parent GetOrSet operation.
  2. Slow/Blocking Operations within the Factory: The ~767ms set to cache level operation (whether slow due to Redis, large data, or client-side blocking) ties up worker threads.
  3. Blocking Code: Potential use of synchronous waits (.Result, .Wait()) within the async factory method.

This ties up available worker threads, preventing the application from promptly processing responses from Redis (causing timeouts) or handling other concurrent work, leading to the high QueuedItems count.

Recommendations:

  1. Refactor Profile factory (High Priority):
    • Eliminate Blocking Calls: Ensure async/await is used correctly throughout the factory. Remove all instances of .Result, .Wait(), etc.
    • Simplify Caching Strategy:
      • Recommended: Modify the factory to only fetch data asynchronously from the database. Return the complete data object.
      • Let the outer GetOrSet profile call handle the sole responsibility of caching the object returned by the factory.
      • This avoids nested cache calls, reduces complexity, and minimizes the chance of blocking within the factory.
  2. Investigate Slow Image Cache Set:
    • Determine why caching images takes ~767ms.
    • Check the size of the data being cached. Is it excessive? Can it be optimized (e.g., cache references/URLs instead of large blobs)?
    • Monitor Redis performance during these operations (check SLOWLOG, COMMANDSTATS for SET/HMSET latency, CPU/Memory usage on redis-node-1...).
  3. Monitor Post-Refactoring: Track ThreadPool metrics (QueuedItems, busy threads) and Redis command latency after implementing changes to verify improvement.

@jodydonetti Absolutely, we can work around this, but I’m still trying to reproduce locally to hopefully understand the problem better and hopefully get a minimal repro for you.