cosmo: Cosmo router "Adding connection to epoll - invalid argument" error

Component(s)

router

Component version

router@0.69.1

wgc version

unsure

controlplane version

studio@0.69.0

router version

router@0.69.1

What happened?

I’m trying to make subscriptions work, however I see the cosmo router is logging the following error:

{"level":"error","time":1709293411679,"msg":"Adding connection to epoll","hostname":"cosmo-router-qa-router-6dbdd9c497-bgnsd","pid":1,"component":"@wundergraph/router","service_version":"0.69.1","reqId":"cosmo-router-q 
a-router-6dbdd9c497-bgnsd/TCeWL4cmLc-020819","error":"invalid argument","stacktrace":"github.com/wundergraph/cosmo/router/core.(*WebsocketHandler).handleUpgradeRequest\n\tgithub.com/wundergraph/cosmo/router/core/websoc 
ket.go:264\ngithub.com/wundergraph/cosmo/router/core.(*WebsocketHandler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/core/websocket.go:179\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\tgithub.com/go-chi/chi/v5@v5. 
0.11/mux.go:73\ngithub.com/go-chi/chi/v5.(*Mux).Mount.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/mux.go:315\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\tgithub 
.com/go-chi/chi/v5@v5.0.11/mux.go:443\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/wundergraph/cosmo/router/pkg/cors.(*cors).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/pkg/cors/config.go 
:96\ngithub.com/wundergraph/cosmo/router/internal/requestlogger.(*handler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/internal/requestlogger/requestlogger.go:99\ngithub.com/wundergraph/cosmo/router/pkg/trace.(*Mi 
ddleware).Handler.func1\n\tgithub.com/wundergraph/cosmo/router/pkg/trace/middleware.go:45\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*midd 
leware).serveHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.46.1/handler.go:143\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\tgo.opentelemetry.io/cont 
rib/instrumentation/net/http/otelhttp@v0.46.1/handler.go:81\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.RealIP.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/middleware/re 
alip.go:36\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.RequestID.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/middleware/request_id.go:76\nnet/http.HandlerFunc.ServeHTTP 
\n\tnet/http/server.go:2136\ngithub.com/wundergraph/cosmo/router/internal/recoveryhandler.(*handler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/internal/recoveryhandler/recovery.go:103\ngithub.com/wundergraph/cos 
mo/router/core.(*Router).newServer.func2.1\n\tgithub.com/wundergraph/cosmo/router/core/router.go:836\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\tgithub.com/g 
o-chi/chi/v5@v5.0.11/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2938\nnet/http.(*conn).serve\n\tnet/http/server.go:2009"}

I’d like some guidance in what could this potentially mean

Environment information

Running on a GKE cluster, using helm deployment

Router configuration

# Default values for router.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: wundergraph/cosmo/router
  registry: ghcr.io
  pullPolicy: IfNotPresent
  # -- Overrides the image tag whose default is the chart appVersion.
  version: "latest"

service:
  type: ClusterIP
  port: 3002

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80

serviceAccount:
  # -- Specifies whether a service account should be created
  create: true
  # -- The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: "service-cosmo-qa"

global:
  router:
    enabled: true

  minio:
    enabled: true

  cdn:
    enabled: true
    s3StorageUrl: 'http://minio:changeme@cosmo-minio:9000/cosmo'

  helmTests:
    enabled: false
  seed:
    enabled: false

  studio:
    enabled: false

  clickhouse:
    enabled: false

  postgresql:
    enabled: false

  redis:
    enabled: false

  controlplane:
    enabled: false

  otelcollector:
    enabled: false

  graphqlmetrics:
    enabled: false

  keycloak:
    enabled: false

# Overall Ingress for the Cosmo Platform
ingress:
  enabled: false
  
cdn:
  configuration:
    # This must match the value used in controlplane.configuration.s3StorageUrl
    s3StorageUrl: 'http://minio:changeme@cosmo-qa-minio:9000/cosmo'

router:
  extraEnvVars:
    - name: CDN_URL
      value: "http://cosmo-router-qa-cdn:8787"

    - name: PLAYGROUND_PATH
      value: "/cosmo/v1/"

    - name: GRAPHQL_PATH
      value: "/cosmo/v1/graphql"

  commonConfiguration: |-
    version: "1"
    log_level: "warning"

    headers:
      all: # Header rules for all subgraph requests.
        request:
          - op: "propagate"            # Forward a client header
            named: Authorization       # Exact match (Use the canonicalized version)
          - op: "propagate"            # Forward a client header
            named: session-id       # Exact match (Use the canonicalized version)
    
    engine:
      debug:
        report_websocket_connections: true

  configuration:
    graphApiToken: "SECRET_TOKEN"
    # -- The execution config file to statically configure the router. If set, polling of the config is disabled.
    # -- If your config exceeds 1MB (Kubernetes limit), you have to mount it as a file and set the path in routerConfigPath instead
    executionConfig: ""
    # -- The log level of the router. Default to info if not set.
    logLevel: "debug"
    # -- The URL of the Cosmo Controlplane. Should be internal to the cluster. Default to cloud if not set.
    controlplaneUrl: 'https://cosmocontrolplane.mydomain.com'
    # -- The URL of the Cosmo GraphQL OTEL Collector. Should be internal to the cluster.
    # otelCollectorUrl: 'http://cosmo-qa-otelcollector:4318'
    otelCollectorUrl: 'https://cosmootel.mydomain.com'
    # -- The URL of the Cosmo GraphQL Metrics Collector. Should be internal to the cluster.
    # graphqlMetricsCollectorUrl: 'http://cosmo-qa-graphqlmetrics:4005'
    graphqlMetricsCollectorUrl: 'https://cosmographql.mydomain.com'
    # # -- The URL of the Cosmo CDN. Should be internal to the cluster.
    # cdnUrl: 'http://cosmo-qa-cdn:8787'
    cdnUrl: 'http://cosmo-router-qa-cdn:8787'
    # -- The path to the router execution config file. Before, you have to mount the file as a volume and set the path here.
    # -- A possible to solution could be to use an init container to download the file from a CDN. If set, polling of the config is disabled.
    routerConfigPath: ""

Router execution config

Cannot exec into router pod, it boots me instantly for some reason. How should I get this?

Log output

{"level":"error","time":1709293411679,"msg":"Adding connection to epoll","hostname":"cosmo-router-qa-router-6dbdd9c497-bgnsd","pid":1,"component":"@wundergraph/router","service_version":"0.69.1","reqId":"cosmo-router-q 
a-router-6dbdd9c497-bgnsd/TCeWL4cmLc-020819","error":"invalid argument","stacktrace":"github.com/wundergraph/cosmo/router/core.(*WebsocketHandler).handleUpgradeRequest\n\tgithub.com/wundergraph/cosmo/router/core/websoc 
ket.go:264\ngithub.com/wundergraph/cosmo/router/core.(*WebsocketHandler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/core/websocket.go:179\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\tgithub.com/go-chi/chi/v5@v5. 
0.11/mux.go:73\ngithub.com/go-chi/chi/v5.(*Mux).Mount.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/mux.go:315\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP\n\tgithub 
.com/go-chi/chi/v5@v5.0.11/mux.go:443\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/wundergraph/cosmo/router/pkg/cors.(*cors).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/pkg/cors/config.go 
:96\ngithub.com/wundergraph/cosmo/router/internal/requestlogger.(*handler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/internal/requestlogger/requestlogger.go:99\ngithub.com/wundergraph/cosmo/router/pkg/trace.(*Mi 
ddleware).Handler.func1\n\tgithub.com/wundergraph/cosmo/router/pkg/trace/middleware.go:45\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*midd 
leware).serveHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.46.1/handler.go:143\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\tgo.opentelemetry.io/cont 
rib/instrumentation/net/http/otelhttp@v0.46.1/handler.go:81\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.RealIP.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/middleware/re 
alip.go:36\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5/middleware.RequestID.func1\n\tgithub.com/go-chi/chi/v5@v5.0.11/middleware/request_id.go:76\nnet/http.HandlerFunc.ServeHTTP 
\n\tnet/http/server.go:2136\ngithub.com/wundergraph/cosmo/router/internal/recoveryhandler.(*handler).ServeHTTP\n\tgithub.com/wundergraph/cosmo/router/internal/recoveryhandler/recovery.go:103\ngithub.com/wundergraph/cos 
mo/router/core.(*Router).newServer.func2.1\n\tgithub.com/wundergraph/cosmo/router/core/router.go:836\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\tgithub.com/g 
o-chi/chi/v5@v5.0.11/mux.go:90\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2938\nnet/http.(*conn).serve\n\tnet/http/server.go:2009"}

Additional context

We have router + CDN in one cluster, and studio, controlplane, etc in another.

Everything can connect to itself as that has been tested, and all pods are running fine.

The way we set it up is: the LB does header based route mapping. If the “upgrade” header is present, means its a websocket, and goes directly to the cosmo router pod. Else it goes to our spring api-gateway.

I would like to understand what exactly cosmo router is complaining about so I can investigate further. Thanks

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 33 (16 by maintainers)

Most upvoted comments

We’ve fixed the issue and will release a new version soon.

After encountering the error using Apollo Sandbox, the same issue started appearing with Postman as well. It seems the problem is specifically related to Apollo Sandbox, and for some reason, it then affects all subsequent requests from other clients.