dapr: Need help troubleshooting: "error finding address for actor type with id "

In what area(s)?

Actors /area placement

What version of Dapr?

1.0.0-rc.3

Expected Behavior

I have a service that has 3 actors on it. It works fine using dapr locally but doesn’t work when deployed to Kubernetes. I’ve uninstalled and reinstalled dapr in kubernetes a couple times to be sure. But there isn’t an issue there.

Actual Behavior

The .NET SDK exception when trying to invoke the method:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HM67KBEG6M8G", Request id "0HM67KBEG6M8G:00000002": An unhandled exception was thrown by the application.
      Dapr.DaprApiException: error invoke actor method: error finding address for actor type SolarSystemMiningActor with id 9a97ec50-a6e7-4e8f-86fa-ff649d27ac22
         at Dapr.Actors.DaprHttpInteractor.SendAsyncHandleUnsuccessfulResponse(Func`1 requestFunc, String relativeUri, CancellationToken cancellationToken) in /dotnet-sdk/src/Dapr.Actors/DaprHttpInteractor.cs:line 377
         at Dapr.Actors.DaprHttpInteractor.SendAsync(Func`1 requestFunc, String relativeUri, CancellationToken cancellationToken) in /dotnet-sdk/src/Dapr.Actors/DaprHttpInteractor.cs:line 285
         at Dapr.Actors.DaprHttpInteractor.InvokeActorMethodWithRemotingAsync(ActorMessageSerializersManager serializersManager, IActorRequestMessage remotingRequestRequestMessage, CancellationToken cancellationToken) in /dotnet-sdk/src/Dapr.Actors/DaprHttpInteractor.cs:line 120
         at Dapr.Actors.Communication.Client.ActorRemotingClient.InvokeAsync(IActorRequestMessage remotingRequestMessage, CancellationToken cancellationToken) in /dotnet-sdk/src/Dapr.Actors/Communication/Client/ActorRemotingClient.cs:line 39
         at Dapr.Actors.Client.ActorProxy.InvokeMethodAsync(Int32 interfaceId, Int32 methodId, String methodName, IActorRequestMessageBody requestMsgBodyValue, CancellationToken cancellationToken) in /dotnet-sdk/src/Dapr.Actors/Client/ActorProxy.cs:line 219
         at <StartupCode$Chronos-Mining-Web>.$EventHandlerController.OnAsteroidCreated@75-2.Invoke(Unit unitVar0) in /app/Chronos.Mining/Chronos.Mining.Web/Controllers/EventHandlerController.fs:line 75
         at FSharp.Control.Tasks.TaskBuilder.StepStateMachine`1.nextAwaitable()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
         at Dapr.CloudEventsMiddleware.ProcessBodyAsync(HttpContext httpContext, String charSet) in /dotnet-sdk/src/Dapr.AspNetCore/CloudEventsMiddleware.cs:line 114
         at Chronos.Utilities.RequestResponseLoggingMiddleware.Invoke(HttpContext context) in /app/Chronos.Common.AspNetCore/RequestResponseLoggingMiddleware.cs:line 29
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

The dapr side car’s logs:

time="2021-02-02T17:49:17.9175701Z" level=info msg="starting Dapr Runtime -- version 1.0.0-rc.3 -- commit dbfa3e4" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9176126Z" level=info msg="log level set to: info" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9178261Z" level=info msg="metrics server started on :9090/" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.metrics type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9178779Z" level=info msg="loading default configuration" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9178965Z" level=info msg="kubernetes mode configured" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9179027Z" level=info msg="app id: mining" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9180439Z" level=info msg="mTLS enabled. creating sidecar authenticator" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9181944Z" level=info msg="trust anchors and cert chain extracted successfully" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.security type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9182282Z" level=info msg="authenticator created" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9248038Z" level=info msg="Initialized name resolution to kubernetes" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9269488Z" level=info msg="component loaded. name: kubernetes, type: secretstores.kubernetes" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9333112Z" level=info msg="component loaded. name: redis-state, type: state.redis" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9334424Z" level=info msg="waiting for all outstanding components to be processed" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9366692Z" level=info msg="component loaded. name: messagebus, type: pubsub.redis" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9367448Z" level=info msg="all outstanding components processed" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9367927Z" level=info msg="enabled gRPC tracing middleware" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.grpc.api type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9368141Z" level=info msg="enabled gRPC metrics middleware" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.grpc.api type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9368947Z" level=info msg="API gRPC server is running on port 50001" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9372084Z" level=info msg="enabled metrics http middleware" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.http type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.937241Z" level=info msg="enabled tracing http middleware" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.http type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9372982Z" level=info msg="http server is running on port 3500" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9373292Z" level=info msg="The request body size parameter is: 4" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9373551Z" level=info msg="enabled gRPC tracing middleware" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.grpc.internal type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9373896Z" level=info msg="enabled gRPC metrics middleware" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.grpc.internal type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9374123Z" level=info msg="sending workload csr request to sentry" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.grpc.internal type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9473722Z" level=info msg="certificate signed successfully" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.grpc.internal type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9476425Z" level=info msg="internal gRPC server is running on port 50002" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9476809Z" level=info msg="application protocol: http. waiting on port 80.  This will block until the app is listening on that port." app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9477402Z" level=info msg="starting workload cert expiry watcher. current cert expires on: 2021-02-03 18:04:17 +0000 UTC" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.grpc.internal type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.9481058Z" level=info msg="application discovered on port 80" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.0266006Z" level=info msg="application configuration loaded" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.0267307Z" level=info msg="actor runtime started. actor idle timeout: 1h0m0s. actor scan interval: 30s" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.actor type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.0392252Z" level=info msg="placement tables updated, version: 5" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime.actor.internal.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.0419316Z" level=info msg="app is subscribed to the following topics: [mining-equipment-added metal-capacity-units-set asteroid-created solar-system-created ship-created] through pubsub=messagebus" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.0422876Z" level=warning msg="redis streams: BUSYGROUP Consumer Group name already exists" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.contrib type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.0424681Z" level=warning msg="redis streams: BUSYGROUP Consumer Group name already exists" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.contrib type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.0426284Z" level=warning msg="redis streams: BUSYGROUP Consumer Group name already exists" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.contrib type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.0428844Z" level=warning msg="redis streams: BUSYGROUP Consumer Group name already exists" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.contrib type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.0431132Z" level=warning msg="redis streams: BUSYGROUP Consumer Group name already exists" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.contrib type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.043187Z" level=info msg="dapr initialized. Status: Running. Init Elapsed 125.2846ms" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.4656394Z" level=warning msg="retriable error returned from app while processing pub/sub event d0dface0-e418-4399-9ae6-bba44aa4e5ab: . status code returned: 500" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:18.4657139Z" level=warning msg="retriable error returned from app while processing pub/sub event 14d672a0-e1a9-43f6-b0b8-bb0f477d7fde: . status code returned: 500" app_id=mining instance=mining-664bd89997-cgrhv scope=dapr.runtime type=log ver=1.0.0-rc.3

(The retriable errors are because as a result of consuming a pub/sub message I’m making a call to an Actor)

The dapr placement service logs:

> klo dapr-placement-server-0 -n dapr-system
time="2021-02-02T17:35:55.5036354Z" level=info msg="starting Dapr Placement Service -- version 1.0.0-rc.3 -- commit dbfa3e4" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:35:55.5037436Z" level=info msg="log level set to: info" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:35:55.5039444Z" level=info msg="metrics server started on :9090/" instance=dapr-placement-server-0 scope=dapr.metrics type=log ver=1.0.0-rc.3
time="2021-02-02T17:35:55.5044764Z" level=info msg="Raft server is starting on 127.0.0.1:8201..." instance=dapr-placement-server-0 scope=dapr.placement.raft type=log ver=1.0.0-rc.3
time="2021-02-02T17:35:55.5045819Z" level=info msg="mTLS enabled, getting tls certificates" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:35:55.5046176Z" level=info msg="starting watch for certs on filesystem: /var/run/dapr/credentials" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:35:57.0894178Z" level=info msg="certificates detected" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:35:57.0895313Z" level=info msg="tls certificates loaded successfully" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:35:57.0895651Z" level=info msg="placement service started on port 50005" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:35:57.0896633Z" level=info msg="cluster leadership acquired" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:35:57.0910679Z" level=info msg="Healthz server is listening on :8080" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:35:57.0911645Z" level=info msg="leader is established." instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:47:14.0915367Z" level=info msg="Start desseminating tables. memberUpdateCount: 1, streams: 3, targets: 1, table generation: 1" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:47:14.0923235Z" level=info msg="Completed dessemination. memberUpdateCount: 1, streams: 3, targets: 1, table generation: 1" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:47:19.0916059Z" level=info msg="Start desseminating tables. memberUpdateCount: 1, streams: 10, targets: 2, table generation: 2" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:47:19.0972405Z" level=info msg="Completed dessemination. memberUpdateCount: 1, streams: 10, targets: 2, table generation: 2" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:48:32.091439Z" level=info msg="Start desseminating tables. memberUpdateCount: 2, streams: 0, targets: 0, table generation: 4" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:48:32.0915036Z" level=info msg="Completed dessemination. memberUpdateCount: 2, streams: 0, targets: 0, table generation: 4" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.5914926Z" level=info msg="Start desseminating tables. memberUpdateCount: 1, streams: 9, targets: 1, table generation: 5" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:17.5922274Z" level=info msg="Completed dessemination. memberUpdateCount: 1, streams: 9, targets: 1, table generation: 5" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:20.0915692Z" level=info msg="Start desseminating tables. memberUpdateCount: 1, streams: 13, targets: 2, table generation: 6" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-02T17:49:20.0950564Z" level=info msg="Completed dessemination. memberUpdateCount: 1, streams: 13, targets: 2, table generation: 6" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-03T15:52:10.5032936Z" level=info msg="Start desseminating tables. memberUpdateCount: 8, streams: 13, targets: 2, table generation: 14" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3
time="2021-02-03T15:52:10.5089016Z" level=info msg="Completed dessemination. memberUpdateCount: 8, streams: 13, targets: 2, table generation: 14" instance=dapr-placement-server-0 scope=dapr.placement type=log ver=1.0.0-rc.3

The mining service manifest:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mining
  labels:
    app: mining
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mining
  template:
    metadata:
      labels:
        app: mining
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "mining"
        dapr.io/app-port: "80"
        dapr.io/max-concurrency: "1"
    spec:
      containers:
        - name: web
          image: <my image>
          ports:
            - containerPort: 80
          imagePullPolicy: Always
          env:
            - name: DAPR_HTTP_PORT
              value: "3500"

The dapr components yaml:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: redis-state
spec:
  type: state.redis
  version: v1
  metadata:
  - name: redisHost
    value: {{ .Values.redisServiceName }}
  - name: redisPassword
    value: {{ .Values.redisPassword }}
  - name: actorStateStore
    value: "true"
---
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: messagebus
spec:
  type: pubsub.redis
  version: v1
  metadata:
  - name: redisHost
    value: {{ .Values.redisServiceName }}
  - name: redisPassword
    value: {{ .Values.redisPassword }}

I even see via the “/dapr/config” call that it is indeed returning the 3 actors that it should host:

$ curl localhost:6016/dapr/config
{"entities":["MinerActor","ResourcesActor","SolarSystemMiningActor"]}

If I port forward to daprd in the pod and try calling the actor i get errors as well:

$ curl localhost:3500/v1.0/actors/SolarSystemMiningActor/abc123/state/redis-state
{"errorCode":"ERR_ACTOR_INSTANCE_MISSING","message":"actor instance is missing"}

$ curl localhost:3500/v1.0/actors/SolarSystemMiningActor/9a97ec50-a6e7-4e8f-86fa-ff649d27ac22/method/tickAsync
{"errorCode":"ERR_ACTOR_INVOKE_METHOD","message":"error invoke actor method: error from actor service: "}

My kubectl get pods -n dapr-system:

NAME                                     READY   STATUS    RESTARTS   AGE
dapr-dashboard-6c4b54cc55-n58wv          1/1     Running   0          30h
dapr-operator-79df664bd-vnk5n            1/1     Running   1          30h
dapr-placement-server-0                  1/1     Running   0          30h
dapr-sentry-56df54f4d9-5vn69             1/1     Running   0          30h
dapr-sidecar-injector-56ddf9d5c4-8q2lv   1/1     Running   0          30h

Steps to Reproduce the Problem

I have basically an exact duplicate of another F# project that is working, the only difference is that I have 3 actors in this one opposed to 1 in the working one. All of the nuget packages, startup.fs, and program.fs are exactly the same too but for some reason only this one service’s actors don’t get registered or something where as the other service works fine.

Basically I just need a hand troubleshooting this as I am at a loss on where to go from here. I can’t determine from any of the logs anywhere why this particular project’s actors dont get registered.

Release Note

RELEASE NOTE: