spin: io error: broken pipe when accessing redis-backed KV store

I have an application (tfenster/verified-bluesky) that worked fine on your cloud, but due to the limit of 1024 entries to the KV store, I moved it to Azure (Azure Kubernetes Service, Azure Cache for Redis). Since then, it works for some time (didn’t exactly check, but seems like a few minutes), but then calls to the KV store fail with “io error: broken pipe”. I have narrowed it down to the following piece of code as the easiest repro, but other places accessing the store also fail with the same error:

package main

import (
	"encoding/json"
	"fmt"
	"net/http"
	"strings"

	spinhttp "github.com/fermyon/spin/sdk/go/v2/http"
	"github.com/fermyon/spin/sdk/go/v2/kv"
)

func init() {
	spinhttp.Handle(func(w http.ResponseWriter, r *http.Request) {
		switch r.Method {
		case http.MethodGet:
			fmt.Println("Getting stats")
			store, err := kv.OpenStore("default")
			if err != nil {
				http.Error(w, "Error opening kv store "+err.Error(), http.StatusInternalServerError)
				return
			}
			defer store.Close()

			fmt.Println("Getting keys")
			keys, err := store.GetKeys()
			if err != nil {
				http.Error(w, "Error getting keys from kv store "+err.Error(), http.StatusInternalServerError)
				return
			}
...
		}
	})
}

func main() {}

The call to kv.OpenStore still seems to work, but store.GetKeys() fails and I get this response:

HTTP/1.1 500 Internal Server Error
Date: Sun, 05 Jan 2025 08:23:35 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 55
Connection: close
x-content-type-options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains

Error getting keys from kv store io error: broken pipe

For full context, see main.go. The logs only show this

Getting stats
Getting keys

The setup of the redis backend and creation of the runtime config looks like this

az redis create -n $redisName -g $rgAndClusterName -l $location --sku Basic --vm-size c0 --redis-configuration '{ "maxmemory-policy": "noeviction" }'
$redisKeys = $(az redis list-keys -n $redisName -g $rgAndClusterName) | ConvertFrom-Json

@"
[key_value_store.default]
type = "redis"
url = "rediss://:$($redisKeys.primaryKey)@$redisName.redis.cache.windows.net:6380"
"@ | Out-File ./runtime-config-redis.toml

I then bring that to this secret (content obviously redacted)

apiVersion: v1
kind: Secret
metadata:
  name: verified-bluesky-runtime-config
type: Opaque
data:
  runtime-config.toml: ...

Deploying the spin app looks like this spinapp.yaml and the latest build to create and push the OCI artifact is here.

Let me know if I can share anything else

Not sure if that is relevant, but this is the output of the commands you asked for in the template, taken from my dev environment:

  • Spin version (spin --version):
vscode ➜ /workspaces/verified-bluesky/deployment (main) $ spin --version
spin 3.1.1 (aa919ce 2024-12-20)
  • Installed plugin versions (spin plugins list --installed)
vscode ➜ /workspaces/verified-bluesky/deployment (main) $ spin plugins list --installed
cloud 0.10.0 [installed]
kube 0.3.1 [installed]

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 30 (14 by maintainers)

Commits related to this issue

Most upvoted comments

Looks good! It works now for ~1h, the previous version never made it so long. Thanks for the quick solution and turnaround to everyone involved! @itowlson @ThorstenHans @fibonacci1729 @radu-matei @kate-goldenring

@ThorstenHans that’s what I first tried, but I ran into the RU limits very quickly for the cheap instances, so I would prefer Redis

Did that as well, thanks. A little nit to add in case anyone comes across this: The kubectl annotate command also needed --overwrite for me

Great to hear it is working! We also released the shim with a fix so you can update to shim v0.18.0 as well:

# Remove annotation from all nodes so can reapply it later to trigger a new install of the shim
kubectl annotate node --all kwasm.sh/kwasm-node=-
# Upgrade node installer to latest shim
helm upgrade --install \
  kwasm-operator kwasm/kwasm-operator \
  --namespace kwasm \
  --create-namespace \
  --set "kwasmOperator.installerImage=ghcr.io/spinkube/containerd-shim-spin/node-installer:v0.18.0"
# Reapply annotation to provision all nodes
kubectl annotate node --all kwasm.sh/kwasm-node=true

@tfenster you can use your existing app – no need to rebuild. It will use the newly updated host components of the Spin runtime.

Here is an updated node installer image you can try out. It has the Spin patch in it:

# Remove annotation from all nodes so can reapply it later to trigger a new install of the shim
kubectl annotate node --all kwasm.sh/kwasm-node=-
# Upgrade node installer to latest shim
helm upgrade --install \
  kwasm-operator kwasm/kwasm-operator \
  --namespace kwasm \
  --create-namespace \
  --set "kwasmOperator.installerImage=ghcr.io/spinkube/containerd-shim-spin/node-installer:20250114-015325-gf938c61"
# Reapply annotation to provision all nodes
kubectl annotate node --all kwasm.sh/kwasm-node=true

You should see a new kwasm job start and complete for each node.

@tfenster quick update, @kate-goldenring created a corresponding PR over on containerd-shim-spin to land those changes there soon -> https://github.com/spinkube/containerd-shim-spin/pull/263

That’s great to hear, thanks for the confirmation, @tfenster!

Enjoy your weekend!

Thanks for the hint @radu-matei! I tried and just to confirm your fix, @fibonacci1729, it works locally now for > 1h with the canary build. The latest release would have definitely broken in that time, so looks like it is fixed 🎉

Mentioning that we also have canary builds of Spin, so you could at least try things out locally by grabbing Spin from https://github.com/fermyon/spin/releases.

Hey @tfenster! We will kick off a patch release of Spin Monday after the AM public Spin meeting which should be available by the EOD. I imagine getting a release of the containerd-shim-spin should follow a similar timeline the next day. We’ll try to get you situated by mid week next week!

Thanks a lot @fibonacci1729 for working on it so quickly! Do I now need to wait for a release or can I already somehow get the fix?

First, a new Spin release must be cut, containerd-shim-spin must update the corresponding dependency, and also cut a release. I’ll reach out to corresponding people and try to get a timeframe for those.

Thanks a lot @fibonacci1729 for working on it so quickly! Do I now need to wait for a release or can I already somehow get the fix?

Hey @tfenster! Just context switching to this now. I’ll have a better idea by the EOD once i can reproduce and poke around a bit to confirm the fix. I’ll update here.

ok, then I’ll patiently wait for a fix 😊 Thanks a lot for the quick check!

At a glance, this seems like a quirk of using redis::aio::MultiplexedConnection in the kv redis factor, possibly in conjunction with how the redis instance timeout is configured with defaults. I’ll poke a little deeper to understand what’s going on here.

That’s the trick. After waiting some time I see the same error. I’ll try to spot how / where this happens

cc @ThorstenHans for Azure wisdom and @fibonacci1729 for Redis KV knowledge