spin: io error: broken pipe when accessing redis-backed KV store
I have an application (tfenster/verified-bluesky) that worked fine on your cloud, but due to the limit of 1024 entries to the KV store, I moved it to Azure (Azure Kubernetes Service, Azure Cache for Redis). Since then, it works for some time (didn’t exactly check, but seems like a few minutes), but then calls to the KV store fail with “io error: broken pipe”. I have narrowed it down to the following piece of code as the easiest repro, but other places accessing the store also fail with the same error:
package main
import (
"encoding/json"
"fmt"
"net/http"
"strings"
spinhttp "github.com/fermyon/spin/sdk/go/v2/http"
"github.com/fermyon/spin/sdk/go/v2/kv"
)
func init() {
spinhttp.Handle(func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
fmt.Println("Getting stats")
store, err := kv.OpenStore("default")
if err != nil {
http.Error(w, "Error opening kv store "+err.Error(), http.StatusInternalServerError)
return
}
defer store.Close()
fmt.Println("Getting keys")
keys, err := store.GetKeys()
if err != nil {
http.Error(w, "Error getting keys from kv store "+err.Error(), http.StatusInternalServerError)
return
}
...
}
})
}
func main() {}
The call to kv.OpenStore still seems to work, but store.GetKeys() fails and I get this response:
HTTP/1.1 500 Internal Server Error
Date: Sun, 05 Jan 2025 08:23:35 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 55
Connection: close
x-content-type-options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains
Error getting keys from kv store io error: broken pipe
For full context, see main.go. The logs only show this
Getting stats
Getting keys
The setup of the redis backend and creation of the runtime config looks like this
az redis create -n $redisName -g $rgAndClusterName -l $location --sku Basic --vm-size c0 --redis-configuration '{ "maxmemory-policy": "noeviction" }'
$redisKeys = $(az redis list-keys -n $redisName -g $rgAndClusterName) | ConvertFrom-Json
@"
[key_value_store.default]
type = "redis"
url = "rediss://:$($redisKeys.primaryKey)@$redisName.redis.cache.windows.net:6380"
"@ | Out-File ./runtime-config-redis.toml
I then bring that to this secret (content obviously redacted)
apiVersion: v1
kind: Secret
metadata:
name: verified-bluesky-runtime-config
type: Opaque
data:
runtime-config.toml: ...
Deploying the spin app looks like this spinapp.yaml and the latest build to create and push the OCI artifact is here.
Let me know if I can share anything else
Not sure if that is relevant, but this is the output of the commands you asked for in the template, taken from my dev environment:
- Spin version (
spin --version):
vscode ➜ /workspaces/verified-bluesky/deployment (main) $ spin --version
spin 3.1.1 (aa919ce 2024-12-20)
- Installed plugin versions (
spin plugins list --installed)
vscode ➜ /workspaces/verified-bluesky/deployment (main) $ spin plugins list --installed
cloud 0.10.0 [installed]
kube 0.3.1 [installed]
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 30 (14 by maintainers)
Commits related to this issue
- fix #2974 -- use redis::aio::ConnectionManager Signed-off-by: Brian H <brian.hardock@fermyon.com> — committed to fibonacci1729/spin by fibonacci1729 2 years ago
- fix #2974 -- use redis::aio::ConnectionManager Signed-off-by: Brian H <brian.hardock@fermyon.com> — committed to fibonacci1729/spin by fibonacci1729 2 years ago
- Merge pull request #2976 from fibonacci1729/fix-2974 fix #2974 -- use redis::aio::ConnectionManager — committed to spinframework/spin by fibonacci1729 2 years ago
- fix #2974 -- use redis::aio::ConnectionManager Signed-off-by: Brian H <brian.hardock@fermyon.com> (cherry picked from commit ec11ba216875feefb669105968f018532fbe27d3) — committed to spinframework/spin by fibonacci1729 2 years ago
- Merge pull request #2978 from fermyon/backport-2976-to-v3.1 [Backport v3.1] fix #2974 -- use redis::aio::ConnectionManager — committed to spinframework/spin by fibonacci1729 2 years ago
Looks good! It works now for ~1h, the previous version never made it so long. Thanks for the quick solution and turnaround to everyone involved! @itowlson @ThorstenHans @fibonacci1729 @radu-matei @kate-goldenring
@ThorstenHans that’s what I first tried, but I ran into the RU limits very quickly for the cheap instances, so I would prefer Redis
Did that as well, thanks. A little nit to add in case anyone comes across this: The
kubectl annotatecommand also needed--overwritefor meGreat to hear it is working! We also released the shim with a fix so you can update to shim
v0.18.0as well:@tfenster you can use your existing app – no need to rebuild. It will use the newly updated host components of the Spin runtime.
Here is an updated node installer image you can try out. It has the Spin patch in it:
You should see a new
kwasmjob start and complete for each node.@tfenster quick update, @kate-goldenring created a corresponding PR over on
containerd-shim-spinto land those changes there soon -> https://github.com/spinkube/containerd-shim-spin/pull/263That’s great to hear, thanks for the confirmation, @tfenster!
Enjoy your weekend!
Thanks for the hint @radu-matei! I tried and just to confirm your fix, @fibonacci1729, it works locally now for > 1h with the canary build. The latest release would have definitely broken in that time, so looks like it is fixed 🎉
Mentioning that we also have canary builds of Spin, so you could at least try things out locally by grabbing Spin from https://github.com/fermyon/spin/releases.
Hey @tfenster! We will kick off a patch release of Spin Monday after the AM public Spin meeting which should be available by the EOD. I imagine getting a release of the containerd-shim-spin should follow a similar timeline the next day. We’ll try to get you situated by mid week next week!
First, a new Spin release must be cut, containerd-shim-spin must update the corresponding dependency, and also cut a release. I’ll reach out to corresponding people and try to get a timeframe for those.
Thanks a lot @fibonacci1729 for working on it so quickly! Do I now need to wait for a release or can I already somehow get the fix?
Hey @tfenster! Just context switching to this now. I’ll have a better idea by the EOD once i can reproduce and poke around a bit to confirm the fix. I’ll update here.
ok, then I’ll patiently wait for a fix 😊 Thanks a lot for the quick check!
At a glance, this seems like a quirk of using
redis::aio::MultiplexedConnectionin the kv redis factor, possibly in conjunction with how the redis instance timeout is configured with defaults. I’ll poke a little deeper to understand what’s going on here.That’s the trick. After waiting some time I see the same error. I’ll try to spot how / where this happens
cc @ThorstenHans for Azure wisdom and @fibonacci1729 for Redis KV knowledge