azure-functions-durable-extension: Entity Operation called too many times under load

DF 2.1 Consumption Plan UK South

---

@cgillum I’m doing a load test of Entites and getting some unexpected results.

Given 3 Event types G, N, D

Each event is published to 10,000 entities using the IDurableEntityClient Proxy such like…

client.SignalEntityAsync<IActor>(entityId, proxy => ...

When handling an event inside the Entity, the last operation is to signal to a singleton aggregation Entity which keep a running total of how many instances of each event type has been received.

I expected that when I read back the entity state of the aggregation Entity, it would be :

G = 10,000 N = 10,000 D = 10,000

but what I’m actually seeing is :

G = 10,086 N = 10,111 D = 10,163

---

I’ve added App Insights telemetry to my Entities so I could observe how many times each entity is processing an event by tracking a customEvent as the last operation when handling an event.

The results don’t add up.

My assumption is that the entity state is being persisted, but the control queue message is not being completed, so it may be replayed thus running the method again causing downstream duplication of signals etc. Notice how the events are duplicated on another host

Another thing that I’ve noticed is that the Function App appears to just stop processing for 10 minutes and then resumes. There is no user-code in the App that would explain this.

---

I’ve ran the above test a few times now, I keep getting similar results so its not a one-off.

{
  "version": "2.0",
  "extensions": {
    "durableTask": {
      "hubName": "MessagesTaskHub",
      "storageProvider": {
        "partitionCount": 4
      }
    }
  },
  "logging": {
    "logLevel": {
      "Function": "Warning",
      "Host" : "Warning"
    }
  }
}

Example execution ID of an entity that generated too many events between the time of 12/30/2019, 9:20:11.826 UTC and 12/30/2019, 9:27:11.624 PM

5c720ba8f91b4396912987555ac175db