dapr: Seeing 500 errors from Dapr when restarting pods running Go servers

Problem Statement

When restarting Kubernetes pods running a Go HTTP server (the base net/http package) which are being called by a Dapr service invoke (e.g. http://localhost:3500/v1.0/invoke/{app}/method/{foo}).

There is a window of time where Dapr returns a HTTP 500 as the pods restart. This happens even when there are 2 replicas in the deployment. If you try this with a different platform, e.g. Node.js or Python Flask - then no such issue is seen, no 500s are received at any point.

Notes.

• Adding readiness and liveness probes does not solve the issue.
• Neither does creating a custom HTTP server object, e.g. srv := &http.Server{} with different timeouts
• I noticed Node.js sets Connection: keep-alive header by default, adding this to the Go server made no difference

At this point I don’t know if this is a bug with Dapr, something specific to the Go HTTP implementation or some other issue. Potentially TCP timeouts, keep-alives etc

I’ve created a repo with some simple steps to reproduce this https://github.com/benc-uk/dapr-go-error

The minimal Go HTTP server I’ve used for testing is

package main

import (
	"fmt"
	"log"
	"net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "HELLO there, from %s!", r.URL.Path[1:])
}

func main() {
	http.HandleFunc("/", handler)
	log.Fatal(http.ListenAndServe(":8080", nil))
}