FusionCache: [BUG] Suspicious Duplicated and Commented Code

Describe the bug

After upgrading to the recent version of FusionCache and starting to use the Tags feature, we encountered issues under load. One key symptom is a FusionCacheDistributedCacheException (with an inner TaskCanceledException) when accessing Redis. We also see strange Redis timeouts happening within the .NET portion of our application.

Upon investigating the source code, we noticed suspicious code fragments in both the synchronous and asynchronous implementations of the RedisBackplane subscription logic, where it looks like a block of commented-out code was possibly left by mistake (see links and code snippets in Additional context).

To Reproduce

Unfortunately, we cannot provide a fully isolated Minimal Reproducible Example (MRE) at this time. However, the errors typically arise under heavier loads after upgrading to the recent version of FusionCache that supports the Tags feature.

Expected behavior

We expect that:

  • No FusionCacheDistributedCacheException errors occur under normal or heavy load.
  • The Redis connection timeout issues are minimized or resolved.

Versions

We’ve encountered this issue on:

  • FusionCache version: 2.1.0
  • .NET version: 8.0
  • OS version: Docker in AWS ECS

Screenshots

No relevant screenshots are available.

Additional context

Here are the suspicious sections of code in the RedisBackplane implementations:

Both files show the same pattern:

if (_subscriber is null)
    throw new NullReferenceException("The backplane subscriber is null");

_subscriber.Subscribe(_channel, (_, v) =>
{
    var message = GetMessageFromRedisValue(v, _logger, _subscriptionOptions);

    if (message is null)
        return;

    OnMessage(message);
});

//_subscriber.SubscribeAsync(_channel, (_, v) =>
//{
//    var message = GetMessageFromRedisValue(v, _logger, _subscriptionOptions);
//
//    if (message is null)
//        return;
//
//    OnMessage(message);
//});

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 15 (1 by maintainers)

Most upvoted comments

@angularsen I feel this happening because of "Redis-Sync-Ops": "268", here is not very big number, but can be a thread starvation case.

Right, but in logs it seems like SE.Redis is being used by FusionCache

Ok so maybe it’s the backplane component (made by me and using directly SE.Redis), not the distributed cache component (M.E.C.R).

The only change between a good deploy and one with timeouts, is isolated to upgrading all FusionCache nugets from 1.4.1 to 2.1.0.

Out of curiosity, have you tried 2.0.0?

I could also try other 2.x versions of FusionCache to see how that plays

Maybe just the 2.0.0, the other minor versions are just background noise for this.

Will keep digging as we are a bit blocked by this.

Sorry about it.

Also the strange thing is that a lot of other people are using the latest version without any issue (include big things like HIBP for example).

Btw: I’m about to release the preview for a new version in the next few days, would you be able to use it and see if it goes well?

Thanks!

@jodydonetti I thought so at first too, but if I’m not mistaken I’ve isolated the issue to only upgrading FusionCache nugets from 1.4.1 to 2.1.0, and locking down the SE.Redis version via Directory.Packages.props.

I’m not able to reproduce by upgrading SE.Redis alone. I am able to reproduce by upgrading FusionCache alone, forcing SE.Redis to specific versions. I confirm in logs the exact version of SE.Redis used when deployed.

  • ✅ FusionCache 1.4.1 and StackExchange.Redis 2.8.14
  • ✅ FusionCache 1.4.1 and StackExchange.Redis 2.8.22
  • ✅ FusionCache 1.4.1 and StackExchange.Redis 2.8.31
  • ❌ FusionCache 2.1.0 and StackExchange.Redis 2.8.14 (FusionCache 2.1.0 wants >= 2.8.22 though)
  • ❌ FusionCache 2.1.0 and StackExchange.Redis 2.8.31

I have not yet been able to create a local repro, so I have to deploy to our Dev environment to test each change, which slows down the debugging.

Another observation I’ve made, is that a common denominator for the cache keys that run into timeouts are the few keys I use a short soft factory timeout with, so they often hit a synthetic timeout and continue the factory in the background. The symptom is that these requests frequently run into 5 second timeouts against Redis, which then slows everything down to a halt - I assume because there is no value in memory to return so it has to wait.

I have a few more ideas to try to reproduce locally. Since I suspect soft factory timeout to be involved, I could spam FusionCache with GetOrSet with a very short soft factory timeout to force frequent synthetic timeouts and see if it’s something with the volume of operations combined with synthetic timeouts that provoke this deadlock or possibly thread starvation behavior.

Update: Deployed FC 1.4.1 + SE.Redis 2.8.22 and no timeouts so far.