cosmo: Cosmo router "Adding connection to epoll - invalid argument" error
Component(s)
router
Component version
router@0.69.1
wgc version
unsure
controlplane version
studio@0.69.0
router version
router@0.69.1
What happened?
I’m trying to make subscriptions work, however I see the cosmo router is logging the following error:
{"level":"error","time":1709293411679,"msg":"Adding connection to epoll","hostname":"cosmo-router-qa-router-6dbdd9c497-bgnsd","pid":1,"component":"@wundergraph/router","service_version":"0.69.1","reqId":"cosmo-router-q
a-router-6dbdd9c497-bgnsd/TCeWL4cmLc-020819","error":"invalid argument","stacktrace":"github.com/wundergraph/cosmo/router/core.(*WebsocketHandler).handleUpgradeRequest\n\tgithub.com/wundergraph/cosmo/router/core/websoc
ket.go:264\ngithub.com/wundergraph/cosmo/router/core.(*WebsocketHandler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/core/websocket.go:179\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\tgithub.com/go-chi/chi/v5@v5.
0.11/mux.go:73\ngithub.com/go-chi/chi/v5.(*Mux).Mount.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/mux.go:315\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\tgithub
.com/go-chi/chi/v5@v5.0.11/mux.go:443\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/wundergraph/cosmo/router/pkg/cors.(*cors).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/pkg/cors/config.go
:96\ngithub.com/wundergraph/cosmo/router/internal/requestlogger.(*handler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/internal/requestlogger/requestlogger.go:99\ngithub.com/wundergraph/cosmo/router/pkg/trace.(*Mi
ddleware).Handler.func1\n\tgithub.com/wundergraph/cosmo/router/pkg/trace/middleware.go:45\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*midd
leware).serveHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.46.1/handler.go:143\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\tgo.opentelemetry.io/cont
rib/instrumentation/net/http/otelhttp@v0.46.1/handler.go:81\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.RealIP.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/middleware/re
alip.go:36\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.RequestID.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/middleware/request_id.go:76\nnet/http.HandlerFunc.ServeHTTP
\n\tnet/http/server.go:2136\ngithub.com/wundergraph/cosmo/router/internal/recoveryhandler.(*handler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/internal/recoveryhandler/recovery.go:103\ngithub.com/wundergraph/cos
mo/router/core.(*Router).newServer.func2.1\n\tgithub.com/wundergraph/cosmo/router/core/router.go:836\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\tgithub.com/g
o-chi/chi/v5@v5.0.11/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2938\nnet/http.(*conn).serve\n\tnet/http/server.go:2009"}
I’d like some guidance in what could this potentially mean
Environment information
Running on a GKE cluster, using helm deployment
Router configuration
# Default values for router.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: wundergraph/cosmo/router
registry: ghcr.io
pullPolicy: IfNotPresent
# -- Overrides the image tag whose default is the chart appVersion.
version: "latest"
service:
type: ClusterIP
port: 3002
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
serviceAccount:
# -- Specifies whether a service account should be created
create: true
# -- The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: "service-cosmo-qa"
global:
router:
enabled: true
minio:
enabled: true
cdn:
enabled: true
s3StorageUrl: 'http://minio:changeme@cosmo-minio:9000/cosmo'
helmTests:
enabled: false
seed:
enabled: false
studio:
enabled: false
clickhouse:
enabled: false
postgresql:
enabled: false
redis:
enabled: false
controlplane:
enabled: false
otelcollector:
enabled: false
graphqlmetrics:
enabled: false
keycloak:
enabled: false
# Overall Ingress for the Cosmo Platform
ingress:
enabled: false
cdn:
configuration:
# This must match the value used in controlplane.configuration.s3StorageUrl
s3StorageUrl: 'http://minio:changeme@cosmo-qa-minio:9000/cosmo'
router:
extraEnvVars:
- name: CDN_URL
value: "http://cosmo-router-qa-cdn:8787"
- name: PLAYGROUND_PATH
value: "/cosmo/v1/"
- name: GRAPHQL_PATH
value: "/cosmo/v1/graphql"
commonConfiguration: |-
version: "1"
log_level: "warning"
headers:
all: # Header rules for all subgraph requests.
request:
- op: "propagate" # Forward a client header
named: Authorization # Exact match (Use the canonicalized version)
- op: "propagate" # Forward a client header
named: session-id # Exact match (Use the canonicalized version)
engine:
debug:
report_websocket_connections: true
configuration:
graphApiToken: "SECRET_TOKEN"
# -- The execution config file to statically configure the router. If set, polling of the config is disabled.
# -- If your config exceeds 1MB (Kubernetes limit), you have to mount it as a file and set the path in routerConfigPath instead
executionConfig: ""
# -- The log level of the router. Default to info if not set.
logLevel: "debug"
# -- The URL of the Cosmo Controlplane. Should be internal to the cluster. Default to cloud if not set.
controlplaneUrl: 'https://cosmocontrolplane.mydomain.com'
# -- The URL of the Cosmo GraphQL OTEL Collector. Should be internal to the cluster.
# otelCollectorUrl: 'http://cosmo-qa-otelcollector:4318'
otelCollectorUrl: 'https://cosmootel.mydomain.com'
# -- The URL of the Cosmo GraphQL Metrics Collector. Should be internal to the cluster.
# graphqlMetricsCollectorUrl: 'http://cosmo-qa-graphqlmetrics:4005'
graphqlMetricsCollectorUrl: 'https://cosmographql.mydomain.com'
# # -- The URL of the Cosmo CDN. Should be internal to the cluster.
# cdnUrl: 'http://cosmo-qa-cdn:8787'
cdnUrl: 'http://cosmo-router-qa-cdn:8787'
# -- The path to the router execution config file. Before, you have to mount the file as a volume and set the path here.
# -- A possible to solution could be to use an init container to download the file from a CDN. If set, polling of the config is disabled.
routerConfigPath: ""
Router execution config
Cannot exec into router pod, it boots me instantly for some reason. How should I get this?
Log output
{"level":"error","time":1709293411679,"msg":"Adding connection to epoll","hostname":"cosmo-router-qa-router-6dbdd9c497-bgnsd","pid":1,"component":"@wundergraph/router","service_version":"0.69.1","reqId":"cosmo-router-q
a-router-6dbdd9c497-bgnsd/TCeWL4cmLc-020819","error":"invalid argument","stacktrace":"github.com/wundergraph/cosmo/router/core.(*WebsocketHandler).handleUpgradeRequest\n\tgithub.com/wundergraph/cosmo/router/core/websoc
ket.go:264\ngithub.com/wundergraph/cosmo/router/core.(*WebsocketHandler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/core/websocket.go:179\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\tgithub.com/go-chi/chi/v5@v5.
0.11/mux.go:73\ngithub.com/go-chi/chi/v5.(*Mux).Mount.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/mux.go:315\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\tgithub
.com/go-chi/chi/v5@v5.0.11/mux.go:443\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/wundergraph/cosmo/router/pkg/cors.(*cors).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/pkg/cors/config.go
:96\ngithub.com/wundergraph/cosmo/router/internal/requestlogger.(*handler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/internal/requestlogger/requestlogger.go:99\ngithub.com/wundergraph/cosmo/router/pkg/trace.(*Mi
ddleware).Handler.func1\n\tgithub.com/wundergraph/cosmo/router/pkg/trace/middleware.go:45\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*midd
leware).serveHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.46.1/handler.go:143\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\tgo.opentelemetry.io/cont
rib/instrumentation/net/http/otelhttp@v0.46.1/handler.go:81\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.RealIP.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/middleware/re
alip.go:36\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.RequestID.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/middleware/request_id.go:76\nnet/http.HandlerFunc.ServeHTTP
\n\tnet/http/server.go:2136\ngithub.com/wundergraph/cosmo/router/internal/recoveryhandler.(*handler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/internal/recoveryhandler/recovery.go:103\ngithub.com/wundergraph/cos
mo/router/core.(*Router).newServer.func2.1\n\tgithub.com/wundergraph/cosmo/router/core/router.go:836\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\tgithub.com/g
o-chi/chi/v5@v5.0.11/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2938\nnet/http.(*conn).serve\n\tnet/http/server.go:2009"}
Additional context
We have router + CDN in one cluster, and studio, controlplane, etc in another.
Everything can connect to itself as that has been tested, and all pods are running fine.
The way we set it up is: the LB does header based route mapping. If the “upgrade” header is present, means its a websocket, and goes directly to the cosmo router pod. Else it goes to our spring api-gateway.
I would like to understand what exactly cosmo router is complaining about so I can investigate further. Thanks
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 33 (16 by maintainers)
We’ve fixed the issue and will release a new version soon.
After encountering the error using Apollo Sandbox, the same issue started appearing with Postman as well. It seems the problem is specifically related to Apollo Sandbox, and for some reason, it then affects all subsequent requests from other clients.