dapr: PubSub is failing when non-epoch receiver is not allowed

In what area(s)?

/area runtime

What version of Dapr?

1.8.4

Expected Behavior

Correctly be able to handle events

Actual Behavior

I see the following on the Dapr logs (logging with debug doesn’t provide anything extra)

2022-09-30T16:39:11.321288127Z: [INFO]  time="2022-09-30T16:39:11.321098825Z" level=error msg="error deserializing cloud event in pubsub pubsub-items and topic masked: json: cannot unmarshal array into Go value of type map[string]interface {}" app_id="$Default" instance=4b5fa715e135 scope=dapr.runtime type=log ver=1.8.4

When trying it with the raw payload by specifying rawPayload: true as metadata I can receive messages. Looking at the Subscribe handler it seems that go is unable to UnMarshal the data?

https://github.com/dapr/dapr/blob/e412942cdff68900e89530af56c6989ffabb138f/pkg/runtime/runtime.go#L707

Might this have to do with the fact that I am utilizing / in my keynames? (e.g. "dsmr/reading/electricity_equipment_id":"ID") which is required.

Another thing I noticed is the following but I think this might be unrelated:

MessagingError: At least one receiver for the endpoint is created with epoch of '10', and so non-epoch receiver is not allowed. Either reconnect with a higher epoch, or make sure all epoch receivers are closed or disconnected. TrackingId:9b232d34-0b1d-402a-a7e2-f0b1fd133324_B29, SystemTracker:iothub-ns-masked-masked-fbf06fda96:masked~32766, Timestamp:2022-09-30T16:37:50 Reference:fc2d30b2-4d20-4a98-9bcf-0e32e4d3d4bc, TrackingId:a455603f-c7e7-44b2-ab42-012f8a129088_B29, SystemTracker:iothub-ns-masked-21536972-fbf06fda96:eventhub:iota0dfa5bb~32766|$default, Timestamp:2022-09-30T16:37:50 TrackingId:382612e95832475e943fa22670fee039_G3, SystemTracker:gateway5, Timestamp:2022-09-30T16:37:50

In any case, this is breaking my production environment so I would love to see how I can resolve this together

💡Note: this is one of the events that come in and that break Dapr during the unmarshal step:

{
  data_base64: 'eyJkc21yL3JlYWRpbmcvZWxlY3RyaWNpdHlfZXF1aXBtZW50X2lkIjoiMzE1MzQxNDczMTMxMzAzMDM1MzEzNTM1MzYzNiIsImRzbXIvcmVhZGluZy9nYXNfZXF1aXBtZW50X2lkIjoiVW5rbm93biIsImRzbXIvcmVhZGluZy9lbGVjdHJpY2l0eV90YXJpZmYiOiIwMDAxIiwiZHNtci9yZWFkaW5nL2VsZWN0cmljaXR5X2RlbGl2ZXJlZF8xIjoiNTQuMTciLCJkc21yL3JlYWRpbmcvZWxlY3RyaWNpdHlfcmV0dXJuZWRfMSI6IjMxMC40OCIsImRzbXIvcmVhZGluZy9lbGVjdHJpY2l0eV9kZWxpdmVyZWRfMiI6IjY5Ljk3IiwiZHNtci9yZWFkaW5nL2VsZWN0cmljaXR5X3JldHVybmVkXzIiOiIxMTYuMTIiLCJkc21yL3JlYWRpbmcvZWxlY3RyaWNpdHlfY3VycmVudGx5X2RlbGl2ZXJlZCI6IjAuMDAiLCJkc21yL3JlYWRpbmcvZWxlY3RyaWNpdHlfY3VycmVudGx5X3JldHVybmVkIjoiMC4wMiIsImRzbXIvcmVhZGluZy9waGFzZV9jdXJyZW50bHlfZGVsaXZlcmVkX2wxIjoiMC4wMCIsImRzbXIvcmVhZGluZy9waGFzZV9jdXJyZW50bHlfZGVsaXZlcmVkX2wyIjoiMC4wMCIsImRzbXIvcmVhZGluZy9waGFzZV9jdXJyZW50bHlfZGVsaXZlcmVkX2wzIjoiMC4wMCIsImRzbXIvcmVhZGluZy9waGFzZV9jdXJyZW50bHlfcmV0dXJuZWRfbDEiOiIxNy4wMCIsImRzbXIvcmVhZGluZy9waGFzZV9jdXJyZW50bHlfcmV0dXJuZWRfbDIiOiIwLjAwIiwiZHNtci9yZWFkaW5nL3BoYXNlX2N1cnJlbnRseV9yZXR1cm5lZF9sMyI6IjAuMDAiLCJkc21yL3JlYWRpbmcvcGhhc2Vfdm9sdGFnZV9sMSI6IjIzOS43MCIsImRzbXIvcmVhZGluZy9waGFzZV92b2x0YWdlX2wyIjoiMC4wMCIsImRzbXIvcmVhZGluZy9waGFzZV92b2x0YWdlX2wzIjoiMC4wMCIsImRzbXIvcmVhZGluZy9waGFzZV9wb3dlcl9jdXJyZW50X2wxIjoiMS4wMCIsImRzbXIvcmVhZGluZy9waGFzZV9wb3dlcl9jdXJyZW50X2wyIjoiMC4wMCIsImRzbXIvcmVhZGluZy9waGFzZV9wb3dlcl9jdXJyZW50X2wzIjoiMC4wMCIsImRzbXIvcmVhZGluZy9lbGVjdHJpY2l0eV9ob3VybHlfdXNhZ2UiOiItMC4wMCIsImRzbXIvcmVhZGluZy9nYXNfaG91cmx5X3VzYWdlIjoiMC4wMCIsIiRtZXRhIjp7ImRldmljZVR5cGUiOiJlbGVjdHJpY2l0eS1wMSJ9fQ==',
  datacontenttype: 'application/octet-stream',
  id: 'e1653368-1e43-4dad-89be-3b4729897115',
  pubsubname: 'pubsub-items',
  source: 'Dapr',
  specversion: '1.0',
  topic: 'iota0dfa5bb',
  type: 'com.dapr.event.sent'
}

Steps to Reproduce the Problem

1. Create PubSub with Azure Event Hub
2. Create an epoch receiver? (I think I did this by connecting manually in VS Code with the “Start Monitoring Built-In EventPoint”)
3. Connect with Dapr

Pub Sub YAML

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: pubsub-items
spec:
  type: pubsub.azure.eventhubs
  version: v1
  metadata:
  - name: connectionString
    value: "Endpoint=sb://iothub-ns-masked-masked-masked.servicebus.windows.net/;SharedAccessKeyName=service;SharedAccessKey=masked;EntityPath=masked"
  - name: storageAccountName
    value: "masked"
  - name: storageAccountKey
    value: "masked"
  - name: storageContainerName
    value: "iot-checkpoint"

Release Note

FIX PubSub breaking on error in epoch receiver with Azure Event Hub